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Abstract 

The  need  for  fighter  pilots  in  the  Turkish  Air  Force  is  expected  to  increase  with 
the  planned  acquisition  of  Joint  Strike  Fighters,  ongoing  new  F-16  purchases,  and  other 
upgrades  to  the  Turkish  Air  Force  fighter  inventory.  This  increased  demand  will  affect 
the  current  fighter  training  squadron  curriculum  and  scheduling.  This  study  focuses  on 
issues  related  to  143rd  Oncel  squadron  F-16  Pilot  Training  with  this  projected  increase  in 
number  of  student  pilots  (SP).  Completing  the  training  periods  on  time  is  an  important 
issue  along  with  maintaining  effective  training  performance.  In  our  study  the  Total  Time 
of  the  training  period  serves  as  our  primary  performance  measure.  Simulation  modeling 
concepts  are  applied  to  examine  the  training  period  based  on  the  squadron  syllabus.  After 
constructing  a  simulation  model  using  Arena,  Design  of  experiment,  Regression  Analysis 
and  Metamodeling  are  implemented  to  capture  the  effects  of  the  major  factors,  including 
SP,  Instructor  Pilot,  Bandit  and  F-16,  and  how  they  interact  with  each  other.  The 
utilization  of  IP  and  Bandits  is  also  examined  as  a  performance  measure.  In  addition  we 
conduct  a  sensitivity  analysis  using  our  model  with  the  current  143  Oncel  squadron 


resources. 


Acknowledgments 


I  would  like  to  express  my  sincere  appreciation  to  my  faculty  advisor,  Dr.  J.O. 
Miller,  for  his  guidance  and  support  throughout  the  course  of  this  thesis  effort.  The 
insight  and  experience  was  certainly  appreciated.  I  also  thank  to  Maj.  Mattioda  for  his 
contribution  to  my  thesis  as  a  reader. 

I  would  like  to  thank  my  family  and  my  friend  Gokhan  Altin  for  their  support  and 
latitude  provided  to  me  in  this  endeavor.  I  do  also  owe  my  nation  for  providing  me  the 
chance  to  attend  this  education  program. 


Adem  Okal 


v 


Table  of  Contents 


Page 

Abstract  . iv 

Acknowledgements  . v 

Table  of  Contents  . vi 

List  of  Figures  . viii 

List  of  Tables  . ix 

I.  Introduction  . 1 

Background . 1 

Problem  Statement . 3 

Scope . 4 

Thesis  Organization . 5 

II.  Literature  Review . 6 

Introduction . 6 

Description  of  Flight  Training . 6 

Simulation . 8 

Simulation  Classifications . 9 

Advantages  and  Disadvantages  of  Simulation . 11 

Simulation  Steps . 13 

Arena . 19 

Previous  research . 20 

III.  Methodology . 23 

Introduction . 23 

Assumptions . 23 

Description  of  Flight  Training  in  light  of  Simulation  Model  Concepts . 24 

Conceptual  Model  through  Full  Model . 29 

Basic  sortie  Flow  in  Transition  Phase . 31 

Basic  Sortie  Flow  with  4-ship  Flight . 34 

Main  Model  and  Sortie  Logic . 36 

2-ship  Sortie  in  Full  Model . 36 

4-ship  Sortie  in  Full  Model . 37 

Auxiliary  Process  Models . 39 

Duties  during  Training  Period . 39 

Annual  Leave  for  IPs  and  Bandits . 41 

Daily  Weather  Condition  Checks . 44 

Simulator  Flights . 45 

Sub  Model  for  Statistics . 46 

Attributes  and  Variables . 46 

Verification  and  Validation  of  the  Model . 48 


vi 


Conclusion 


.49 


IV.  Results  and  Analysis . 50 

General . 50 

Design  of  Experiment  for  Simulation  Model . 50 

Metamodeling  and  Regression  Analysis . 58 

Sensitivity  Analysis  for  Recent  Resource  Numbers . 66 

Conclusion . 70 

V.  Conclusions  and  Recommendations . 72 

General . 72 

Research  and  Conclusions . 72 

Recommendations . 74 

Appendix  A.  List  of  Abbreviations . 76 

Appendix  B.  Factorial  Design  with  Coded  Factors . 77 

Appendix  C.  Factors  and  Responses  from  DOE . 78 

Appendix  D.  Main  Effect  Plots  for  IP  and  Bandit  Utilizations . 80 

Appendix  E.  Step  History . 79 

Appendix  F.  Correlations  between  Metamodel  Factors . 81 

Appendix  G.  Process  Analyzer  Results  for  Sensitivity  Analysis . 82 

Bibliography . 83 

Vita . 85 


vii 


List  of  Figures 


Page 

Figure  1.  Ways  to  analyze  a  system . 8 

Figure  2.  A  Seven-Step  Approach  for  Conducting  a  Successful . 13 

Figure  3.  The  Flow  Chart  of  F-16  Pilot  Training  in  Oncel  Squadron . 25 

Figure  4.  System’s  Inputs  and  Outputs . 29 

Figure  5.  2-ship  Sortie  in  the  Conceptual  Model . 31 

Figure  6.  Resource  allocation  for  2-ship  sortie . 32 

Figure  7.  4-ship  Sortie  in  the  Conceptual  Model . 34 

Figure  8.  Allocations  for  4-ship  Sorties . 35 

Figure  9.  Main  Model  in  Full  Model . 36 

Figure  10.  2-ship  Sortie  in  Full  Model . 37 

Figure  11.  4-ship  Sortie  in  Full  Model . 38 

Figure  12.  Design  in  Arena  for  RSU . 40 

Figure  13.  Design  in  Arena  for  SOF . 40 

Figure  14.  Design  in  Arena  for  RTO . 40 

Figure  15.  Sub  model  for  leave  policy  (for  IP  and  Bandit) . 43 

Figure  16.  Sub  model  for  weather  factor . 44 

Figure  17.  Daily  sorties  flown . 46 

Figure  18.  Interaction  Plot  for  Total  Time . 53 

Figure  19.  Main  Effects  Plot  for  Total  Time . 55 

Figure  20.  Cube  Plot  for  Total  Time . 57 

Figure  21.  Normal  Probability  Plot . 61 

Figure  22.  Plot  for  Residuals  vs  Fitted  values . 62 

Figure  23.  Total  Times  of  Training  for  Two  models . 65 


viii 


List  of  Tables 


Page 

Table  1 .  Sortie  distribution  of  Oncel  squadron  . 2 

Table  2.  Operational  Validity  Classification  . 16 

Table  3.  Sortie  Numbers  in  each  phase  and  sub  phase  . 26 

Table  4.  Simulator  missions . 27 

Table  5.  Numbers  of  resources  . 30 

Table  6.  Variables  and  attributes  in  the  model  . 47 

Table  7.  Main  Factors  . 51 

Table  8.  25  Factorial  Design . 51 

Table  9.  Total  Time  Changes  per  Unit  for  Each  Factor . 56 

Table  10.  Variable  selection  via  Stepwise  Technique  . 60 

Table  11.  Higher  Residuals  in  DOE  Matrix . 62 

Table  12.  Analysis  of  Variance  for  Regression  Analysis  . 63 

Table  13.  Parameter  coefficients . 64 

Table  14.  Results  from  the  simulation  model  and  the  metamodel . 65 

Table  15.  Numbers  of  Resources . 66 

Table  16.  Utilizations  depending  on  resource  levels  . 67 

Table  17.  Effects  of  Each  Additional  SP . 68 

Table  18.  Effect  of  Each  Reduction  in  F-16  C . 69 

Table  19.  Effect  of  Each  Reduction  in  F-16  D . 70 


IX 


SIMULATION  MODELING  AND  ANALYSIS  OF  F-16  PILOT  TRAINING 


SQUADRON 


I.  Introduction 


1.  Background 

Turkey  has  a  significant  military  superiority  over  the  Southeastern  region  as  a 
NATO  member.  Certainly  airpower  is  the  leading  factor  to  maintain  this  current  military 
presence.  When  one  visualizes  airpower,  the  first  thing  that  comes  to  mind  is  flight. 
Beside  new  generation  aircraft,  the  human  factor  arises  as  a  key  factor  which  has  a  direct 
relation  with  airpower  sustainability.  An  Air  Force  with  many  kinds  of  modem  aircraft 
without  well-trained  pilots  would  not  have  the  ideal  effectiveness.  Beside  ground  support, 
it  is  obvious  that  aerial  performance  depends  on  how  successful  pilots  handle  their 
assigned  missions  as  well  as  what  type  of  modem  aircraft  they  fly.  Performance  can  be 
monitored  and  improved  by  assessing  the  quality  of  the  flight  courses  and  how 
successfully  pilots  complete  these  courses. 

In  Turkey,  there  are  several  flight  training  schools  for  basic,  intermediate  and 
professional  phases.  The  only  squadron  is  the  143  squadron,  also  called  Oncel  squadron 
stationed  at  4th  Main  Jet  Base,  Akinci.  It  is  the  only  squadron  where  all  F-16  fighter  pilots 
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are  trained  through  the  B  course  which  undergraduate  pilots  take  to  become  F-16  pilots. 
In  addition,  the  squadron  administers  a  refresher  course  for  pilots  who  have  been  away 
from  active  duty  for  a  certain  time.  The  goal  of  the  B  course,  which  is  the  main  focus  of 
this  thesis,  is  to  train  F-16  pilots  with  basic  proficiency  in  F-16  air-to-air  and  air-to- 
surface  mission  tasks.  The  graduate  pilot  is  considered  to  have  fulfilled  all  requirements 
as  a  combat  ready  pilot  except  for  orientation  training  at  his  or  her  assigned  squadron. 

The  B  course  is  the  main  focus  of  Oncel’s  mission.  The  B  course  is  a  62-sortie 
flight  training  program  that  consists  of  three  different  flying  phases  shown  in  Table  1. 

The  first  phase  of  the  B  course  is  called  Basic  Training,  which  is  comprised  of  two  sub 
phases,  basic  training  (BTR)  and  basic  instrument  (BIF).  The  Air-to-Air  (AA)  phase  is 
the  second  phase  of  the  B  course  and  consists  of  basic  fighter  maneuver  (BFM),  air 
combat  maneuvering  (ACM)  and  basic  intercept  (BI)  sub  phases.  The  third  and  final 
phase  of  the  B  course  is  the  Air-to-Ground  (AG)  phase.  The  AG  phase  is  comprised  of 
six  sub  phases.  At  the  end  of  the  AG  phase,  the  Instrument  qualification  (IQ)  and  Tactical 
qualification  (TQ)  check  rides  must  be  accomplished  to  be  able  to  graduate  successfully 
from  the  B  course  (143rd  Training  Syllabus,  2003). 


Table  1:  Sortie  distribution  of  Oncel  squadron 


TRANSITION 

AA 

AG 

NIGHT 

CHECKS 

BTR 

INTERCEPT 

BFM 

ACM 

SA 

SAT 

NTR 

TQ 

IQ 

TOTAL 

SORTIE 

9 

7 

13 

9 

8 

11 

3 

1 

1 

62 
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In  addition  to  these  flight  phases,  academic  and  device  training  are  scheduled  as  a 
supporting  part  before  or  during  the  B  course.  The  time  constraint  to  finish  this  process 
successfully  is  140  weekdays  based  on  a  class  of  20  Student  Pilots  (SPs).  Of  the  140 
weekdays,  35  are  shared  for  academic,  simulator  and  device  training,  and  the  remaining 
105  for  flight  training. 

In  every  training  period  approximately  20-25  student  pilots  attend  B  course.  The 
daily  number  of  sorties  for  the  Oncel  squadron  is  approximately  40,  corresponding  to  20 
mission  sorties,  which  shows  that  most  missions  are  accomplished  with  more  than  one 
aircraft. 

The  pilots  other  than  student  pilots  are  classified  as  instructor  pilots  (IPs)  and 
Bandits  who  fly  as  a  support  unit.  The  group  for  Bandits  contains  all  the  non-student 
pilots  in  the  squadron.  Bandits  are  required  to  fly  certain  assigned  missions  before 
becoming  IP.  Bandits  fly  support  flight,  which  are  required  for  multi-aircraft  mission. 
Besides  these  support  flights,  Bandits  also  are  required  to  fly  a  certain  number  of  sorties 
according  to  regulations  to  maintain  their  currency  for  various  missions. 

1.2.  Problem  Statement 

Turkey  has  a  big  role  in  the  NATO  environment  as  an  air  force  power  over  the 
region.  To  maintain  this  effectiveness  and  to  meet  the  increasing  needs  for  air  force 
power,  Turkey  is  giving  more  importance  to  aviation.  Over  the  next  few  years,  the 
Turkish  Air  Force  is  planning  to  buy  more  F-16s  to  update  their  current  inventory  and 
replace  lost  aircraft.  Turkey  is  also  one  of  the  contractor  countries  of  the  Joint  Strike 
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Fighter  (JSF)  project,  and  plans  to  add  this  aircraft  to  their  inventory  by  2013.  This  new 
aircraft  will  require  pilot  transitions  from  F-16  and  F-4  squadrons,  while  still  maintaining 
the  required  number  of  pilots  in  all  fighter  squadrons.  This  increased  demand  for  pilots 
will  require  a  larger  number  of  student  pilots  enrolled  in  each  class  for  the  143rd  Oncel 
squadron.  To  complete  training  within  the  same  time  frame  with  an  increased  number  of 
student  pilots,  Oncel  needs  to  come  up  with  some  predictions  to  meet  these  demands 
through  additional  IPs,  Bandits  and  aircraft. 

1.3.  Scope 

The  scope  of  this  thesis  research  is  analyze  the  B  course  in  detail  to  understand 
the  effects  of  changing  defined  variables,  including  number  of  SPs,  IPs,  Bandits  and 
aircraft.  This  analysis  includes  construction  of  a  detailed  computer  simulation  of  the 
entire  training  phase  using  the  Arena  discrete  event  simulation  software  package.  The 
model  is  used  to  explore  major  factors  that  affect  the  training  process  to  include  the  time 
duration  of  the  training,  the  number  of  SPs,  IPs,  Bandits  and  aircraft.  This  analysis 
provides  insight  to  see  how  many  additional  weekdays,  IPs  or  Bandits  will  be  needed  to 
have  a  certain  number  of  SPs  graduate  from  the  Oncel  squadron.  With  this  study  it  is 
proposed  to  have  an  idea  about  which  major  factor  or  factors  are  affecting  the  duration  of 
flight  training  period. 
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1.4.  Thesis  Organization 

In  this  chapter  the  main  concern  is  spent  on  providing  a  background  about  the  F- 
16  pilot  training  process  and  its  sub  phases.  Chapter  2  presents  a  literature  review  about 
the  tools  used  to  analyze  this  problem.  Additionally,  a  discussion  is  provided  on  why 
simulation  was  chosen  as  a  research  tool.  Chapter  3  give  details  about  modeling  the  real 
system  by  using  Arena.  Chapter  4  presents  the  outcomes  of  the  simulation  and  the 
analysis.  Chapter  5  summarizes  all  the  phases  of  this  research,  contributions,  and  also 
recommendations  for  further  research. 
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II.  Literature  review 


Introduction 

This  chapter  first  describes  the  training  process  in  the  Oncel  squadron,  and 
secondly  covers  the  topics  related  to  simulation  building,  advantages  and  disadvantages 
of  simulation.  Also  Arena,  a  simulation  software  package,  is  briefly  discussed  in  this 
chapter.  Finally  previous  research  is  presented  at  the  end  of  the  literature  review. 

2.1.  Description  of  Flight  Training 

The  F-16  Pilot  Training  in  Oncel  squadron  consists  of  three  phases,  BTR,  AA, 
and  AG.  There  are  two  training  terms  in  each  year.  During  these  terms,  a  certain  number 
of  SPs  attend  the  flight  training  course.  From  a  modeling  perspective,  each  flight  training 
term  can  be  thought  of  as  a  production  line  in  a  factory  consisting  of  several  stations  or  a 
multi-stage  flow  shop  line.  In  the  conceptual  model,  a  SP  acts  as  an  entity  in  the  system. 
Every  SP  enters  into  the  training  process,  goes  through  all  sub  phases  consecutively,  and 
at  the  end  of  the  training  graduates  by  accomplishing  all  sorties  as  planned.  In  this  flight 
training  process,  there  are  some  primary  constraints  including  the  number  of  IPs,  Bandits, 
aircraft,  and  daily  sortie  blocs.  Beside  these  constraints,  meteorological  conditions, 
runway  availability,  aircraft  maintenance  can  be  assessed  as  secondary  constraints. 
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Every  graduation  date  is  determined  before  the  training  term  starts.  If  maintaining 
a  high  quality  training  program  is  the  primary  objective  of  Oncel  squadron,  then  having 
all  SPs  graduate  on  time  becomes  a  secondary  objective.  For  most  of  the  training  terms, 
the  numbers  of  the  IPs,  Bandits  and  aircraft  usually  show  little  fluctuation.  On  the  other 
hand,  the  number  of  SPs  may  change  significantly  depending  on  the  number  of  SPs 
graduating  from  the  basic  flight  course.  As  a  result,  the  number  of  SPs  for  each  IP  and 
Bandit  increase  as  the  number  of  SPs  increases  if  the  numbers  of  IPs  and  Bandits  do  not 
change  during  each  training  term.  This  fact  means  that  IPs  and  Bandits  fly  more  or  less 
sorties  depending  on  the  number  of  SPs. 

It  is  proposed  that  a  model  can  be  built  to  see  how  the  system  reacts  if  there 
occurs  a  significant  increase  in  the  number  of  SPs.  The  analysis  of  the  interactions 
between  system  variables  like  numbers  of  IPs,  Bandits,  SPs  and  aircraft,  and  the  effect  on 
system  performance  (Total  Time  of  SPs  in  training)  is  the  main  concern  of  this  research. 
There  are  several  approaches  using  common  operations  research  tools  to  analyze  this 
problem.  These  approaches  include  linear  programming,  integer  programming, 
scheduling  theory,  and  simulation.  These  approaches  fall  under  mathematical  models 
shown  at  Figure  1 .  Simulation  modeling  is  chosen  as  the  approach  for  this  research. 

For  an  analyst,  completing  a  study  in  a  faster  and  more  efficient  manner  is 
generally  preferred.  Up  front,  the  analyst  must  decide  which  solution  technique  to 
employ  for  the  problem  at  hand.  Definitely  time  is  a  key  factor  in  determining  an 
appropriate  solution  method  as  well  as  other  resource  constraints  such  as  costs  and 
computer  hardware  and  software  availability. 
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Figure  1.  Techniques  to  analyze  a  system  (Law,  2007:  4) 


What  makes  simulation  preferable  and  more  efficient  over  other  methods  is  a 
question  which  has  a  subjective  nature  as  well.  The  following  section  of  this  chapter 
discusses  simulation  in  more  detail. 

2.2  Simulation 

“Simulation  refers  to  a  broad  combination  of  techniques  and  applications  to 
imitate  the  real  systems.  Simulation  is  usually  implemented  on  computer  by  using  several 
available  software  tools.  Furthermore,  it  can  be  said  that  simulation  is  a  general 
expression  since  the  idea  finds  its  applications  across  many  fields  and  industries” 

(Kelton,  Sadowski  and  Sturrock,  2007:1).  Simulation  deals  with  systems  and  models  of 
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systems.  Simulation  can  be  described  as  a  depiction  of  real  world  systems  in  a  computer 
generated  environment,  often  times  with  a  high  quality  visual  representation. 

To  answer  the  question  of  what  kinds  of  systems  can  be  modeled,  some  examples 
from  daily  life  follow: 

-  A  freeway  system  of  road  intersections,  controls,  and  traffic. 

-  A  supermarket  with  inventory  control,  checkout,  and  customer  service. 

-  An  emergency  facility  in  a  hospital,  including  personnel,  rooms,  equipment,  and 
patient  transport. 

In  the  area  of  military  applications: 

-  A  military  depot  handling  maintenance  supplies. 

-  A  transportation  processes  between  bases,  deployment  process. 

-  Airlift  operations  in  a  certain  region. 

2.2.1.  Simulation  Classifications 

Simulations  are  traditionally  grouped  in  three  classifications  as  discussed  below, 
although  there  are  many  various  classifications  for  simulation  in  the  literature.  A  static 
model  is  defined  as  a  depiction  of  a  system  at  a  specific  time.  If  time  does  not  play  a  role 
in  the  system’s  nature,  then  the  system  can  be  modeled  statically.  On  the  other  hand,  if 
time  plays  a  role  in  the  system’s  processes,  then  it  can  be  modeled  dynamically.  A 
dynamic  model  is  a  representation  of  a  system  as  time  changes.  Monte  Carlo  models  are 
good  examples  for  static  simulation  modeling.  Our  F-16  pilot  training  squadron  is  a  good 
example  for  dynamic  simulation  modeling. 
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A  model  is  called  deterministic  if  there  are  no  random  inputs  to  the  system.  In  this 
category,  a  system’s  response  is  known  for  sure  with  a  given  set  of  deterministic  inputs  to 
the  model.  If  a  system  is  simple  enough  and  does  not  have  a  stochastic  nature,  other 
techniques,  like  differential  equations  or  linear  programming,  are  more  appropriate  than 
simulation  modeling.  On  the  other  hand,  if  a  system  has  randomness  in  the  nature  of 
either  its  inputs  or  processes,  then  a  system’s  model  is  called  stochastic.  A  stochastic 
simulation  model  has  at  least  one  random  input  or  process  which  results  in  random 
outputs.  These  random  outputs  produce  results  that  approximate  the  true  system 
performance  and  are  one  disadvantage  of  simulation  models. 

If  the  state  variables  of  the  system  change  continuously  over  time,  then  the  model 
is  described  as  a  continuous  simulation  model.  Density  of  airflow  through  an  engine 
compressor,  water  levels  of  a  dam,  and  traffic  density  on  a  highway  are  some  examples 
of  a  continuous  simulation  model.  On  the  other  hand,  for  a  discrete  simulation  model,  the 
state  variables  of  the  system  change  only  at  discrete  points  in  time.  Inventory  level  of  a 
military  depot,  emergency  room  patients  in  a  hospital,  and  number  of  customers  in  line 
for  checkout  at  a  grocery  are  some  examples  of  a  discrete  simulation  model.  It  is  possible 
to  model  a  system  containing  both  continuous  and  discrete  features.  Such  models  are 
called  mixed  continuous-discrete  simulation  models.  “Practically  few  systems  are  totally 
discrete  or  continuous,  however  it  is  generally  possible  to  define  any  system  as  either 
discrete  or  continuous  since  one  of  the  classifications  is  superior  to  the  other  in  most 
systems”  (Law  and  Kelton,  2000:72). 
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2.2.2.  Advantages  and  Disadvantages  of  Simulation 

Simulation  modeling  is  currently  being  used  in  many  fields  and  becoming  more 
common  day  by  day.  There  are  lots  of  ongoing  practical  application  and  theoretical 
research  generating  new  aspects  of  simulation.  Here  in  this  section,  some  advantages 
which  made  simulation  a  prominent  tool  are  listed  below. 

1.  Simulation  is  relatively  efficient  in  modeling  complex  systems.  Most  complex 
systems  can  not  be  analyzed  via  analytical  approaches. 

2.  Simulation  provides  a  tool  for  analyzing  new  policies  and  procedures  without 
changing  the  real  systems. 

3.  Simulation  is  an  efficient  tool  to  make  an  investigation  for  non-existing 
systems,  vehicles,  and  so  on.  It  provides  a  means  for  assessing  all  possible 
conditions  and  policies  which  may  be  significant  factors  in  a  system. 

4.  Comparisons  between  alternative  models  and  policies  can  be  done  more 
effectively  via  simulation. 

5.  Simulation  modeling  provides  not  only  numerical  measures  of  system 
performance,  but  also  provides  insight  into  system  performance. 

6.  Simulation  with  animation  features  aids  verification  and  validation  efforts. 

In  every  circumstance,  it  is  not  logical  to  select  simulation  modeling  as  the 

analysis  approach.  Beside  its  advantages,  there  are  some  disadvantages  which  should  be 
considered  as  well. 
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1 .  Simulation  modeling  and  analysis  generally  are  expensive  and  time  consuming. 
Even  with  a  significant  level  of  effort,  a  simulation  study  may  provide  an 
insufficient  analysis  and  meaningless  results. 

2.  Simulation  does  not  give  an  exact  solution,  it  is  just  an  approximation. 

3.  Simulation  outputs  are  difficult  to  interpret.  Due  to  multiple  random  inputs,  it 
may  be  hard  to  distinguish  what  model  factors  affect  specific  outputs  and  how. 

4.  Simulation  requires  special  training.  There  are  many  ways  to  build  a  model  via 
simulation,  and  it  depends  on  a  modeler’s  perspective  to  come  up  with  an 
efficient  model  (Banks,  Carson,  Nelson  and  Nicol,  2006). 

2.3.  Simulation  Steps 

Simulation  has  seven  steps  to  conduct  a  successful  study  which  is  called  a  “seven- 
step  approach”  (Figure  2)  (Law,  2003:66).  The  first  step  in  building  a  simulation  model  is 
to  state  the  problem  precisely.  The  problem  should  be  initially  expressed  and  stated  in 
detail.  Every  question  about  the  system  to  be  included  should  be  clear  before  going 
through  the  next  step  of  the  simulation  modeling.  Even  with  a  good  starting  problem 
formulation,  further  discussions  with  the  decision  maker  and  Subject  Matter  Experts 
(SME)  about  the  system  should  be  ongoing  to  get  more  information  to  model  the  system 
as  accurately  as  possible.  A  Type  III  error,  solving  the  wrong  problem,  can  be 
eliminated  by  understanding  the  problem  and  the  decision  maker’s  point  of  view  clearly. 
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Figure  2:  A  Seven-Step  Approach  for  Conducting  a  Successful 
Simulation  Study  (Law,  2003:67) 


Data  collection  is  needed  to  estimate  the  model  input  data.  More  accurate  and 
precise  inputs  contribute  to  a  more  valid  model,  better  suited  to  address  the  identified 
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problem.  In  general,  it  is  difficult  to  find  the  required  data  due  to  several  reasons  as  listed 
below: 

1 .  Data  not  available 

2.  Data  censored 

3.  Not  enough  data 

4.  Wrong  data 

5.  Cost 

For  the  case  of  having  data  in  hand,  the  task  is  then  to  fit  the  data  for  a  particular 
process  through  the  use  of  a  theoretical  or  empirical  distribution.  Many  software 
packages  have  emerged  for  analyzing  raw  data  and  identifying  appropriate  probability 
distributions  to  use  in  a  simulation  model.  Some  of  these  tools  are  Bestfit,  Input  Analysis 
of  Arena,  Expertfit,  etc. 

After  stating  the  problem  correctly  and  collecting  requisite  data,  it  is  time  to 
construct  a  model  using  a  commercial  simulation  package  like  Arena,  Promodel,  GPSS, 
or  using  general  purpose  computer  languages  like  Visual  Basic  or  JAVA.  Choosing  the 
appropriate  modeling  software  depends  on  many  factors  including  the  analyst’s 
preference,  cost,  availability  of  the  program,  and  time  available  to  conduct  the  study. 

The  purpose  of  model  verification  is  to  ensure  that  the  model  is  constructed 
(coded)  correctly.  In  other  words,  model  verification  assesses  how  well  the  model  meets 
its  specifications  and  implements  what  it  is  expected  to  do.  In  most  cases,  animation  can 
be  used  as  a  tool  to  aid  in  verification. 
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There  are  various  techniques  which  can  be  used  to  verify  the  model.  While 
building  a  large  and  complex  model,  it  is  suggested  that  another  person  review  the  model 
for  verification.  This  technique  is  called  a  structured  walk-through  of  the  program. 
Another  technique  involves  running  the  simulation  with  different  input  data  and  checking 
the  outputs  to  see  whether  they  are  reasonable.  Most  simulation  software  packages  have 
some  debugging  capabilities.  One  common  feature  is  called  a  trace.  A  trace  provides 
detailed  output  for  state  variable  changes  and  other  user  selected  variables  in  the  model. 

“Validation  is  the  process  of  reaching  an  acceptable  level  of  confidence  that  the 
inferences  drawn  from  a  simulation  are  correct  and  applicable  to  the  real-world  system 
being  represented”  (Shannon,  1998:  11).  Validation  gives  an  insight  to  confirm  that  a 
model  is  a  reasonable  representation  of  the  real  system.  It  is  not  a  one-time  check,  but  an 
iterative  process  to  ensure  the  accuracy  of  the  model  throughout  a  simulation  study. 

The  first  step  from  Figure  2  where  validation  is  specifically  mentioned  is  for 
conceptual  model  validation.  In  a  conceptual  model,  the  basic  logic,  structure,  and 
interactions  are  depicted  to  understand  whether  the  problem  is  captured  correctly. 
Conceptual  model  validity  considers  whether  (1)  the  theories  and  assumptions  underlying 
the  conceptual  model  are  correct  and  if  (2)  the  model’s  representation  of  the  problem 
entities,  the  model’s  structure,  logic,  and  mathematical  and  causal  relationships  are 
reasonable  for  the  intended  purpose  of  the  model  (Sargent,  2005:132). 

Operational  validation  is  determining  whether  the  simulation 
model’s  output  behavior  has  the  accuracy  required  for  the  model’s 
intended  purpose  over  the  domain  of  the  model’s  intended  applicability. 

This  is  where  much  of  the  validation  testing  and  evaluation  takes  place. 
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Since  the  complete  simulation  model  is  used  in  operational  validation, 
any  deficiencies  found  may  be  caused  by  errors  introduced  in  any  of  the 
steps  that  are  involved  in  developing  the  simulation  model  (Sargent, 
2005:136). 


For  operational  validation,  outputs  from  a  model  and  a  real  system  are  compared 
by  using  graphical  techniques  to  make  a  subjective  decision,  or  through  formal  statistical 
techniques  such  as  hypothesis  tests  or  confidence  interval  to  make  an  objective  decision. 
Table  1  shows  some  techniques  for  each  approach. 


Table  2:  Operational  Validity  Classification  (Sargent,  2005:136) 


Observable 

System 

Non- observable 
System 

Subjective 

Approach 

*  Comparison  Using 
Graphical  Displays 

*  Explore  Model 

Behavior 

*  Explore  Model 
Behavior 

*  Comparison  to 

Other  Models 

Objective 

Approach 

*  Comparison  Using 
Statistical  Tests 
and  Procedures 

*  Comparison  to 

Other  Models 

Using  Statistical 
Tests 

Various  validation  techniques  are  discussed  next.  One  of  the  techniques  or  any 
combination  can  be  used  for  validation  purposes.  The  choice  of  any  techniques  depends 
on  the  model  developer.  Generally  some  combination  of  techniques  is  preferable. 
Selection  of  techniques  may  be  limited  by  time,  cost,  and  other  resources. 

Historical  Data  Validation:  After  building  the  model,  the  model’s  validation  can 
be  done  by  using  historical  data  if  available.  Historical  data  also  can  be  used  as  input  data 
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into  the  model.  By  running  the  system  with  given  input  data,  the  model  outputs  can  be 
compared  with  the  historical  system  output  to  see  if  the  model  acts  as  the  real  system 
acts. 

Comparison  to  Other  Models:  The  results  from  the  current  model  are  compared 
with  other  model’s  results  which  had  been  assessed  as  valid  before. 

Extreme  Condition  Test :  This  technique  checks  the  model’s  reaction  under 
extreme  conditions.  These  extreme  conditions  include  input  data  like  assigning  arrivals  or 
inventory  level  as  zero.  As  a  rule  of  thumb,  the  output  should  be  reasonable 
corresponding  to  the  extreme  input  data. 

Face  Validity:  Having  the  model  checked  by  another  person,  as  mentioned  in 
verification  section  above  in  this  chapter,  is  another  technique  for  validation.  This  person 
should  be  a  user  of  the  system  or  someone  with  knowledge  of  how  the  system  operates. 

Turing  Tests :  The  outputs  from  a  real  system  and  a  model  are  collected  as 
validation  tools.  Outputs  from  both  sources  are  mixed  and  given  to  a  SME  or  a  well- 
knowledgeable  user.  If  the  SME  or  user  discriminates  between  model  and  real  system 
outputs,  it  shows  that  the  model  is  not  valid. 

Designing  and  Conducting  Simulation  Experiments: 

Before  conducting  simulation  experiments,  the  analyst  must  decide  a  number  of 

issues: 

1 .  The  input  parameters  to  be  varied,  their  range  and  legitimate  combinations, 

2.  Model  run  length  (how  long  to  run  the  simulation) 

3.  Number  of  replications  (Carson,  2003:21-22) 
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All  these  issues  depend  on  various  factors  like  the  amount  of 
data  at  hand,  the  deadline  for  modeling,  and  the  resources.  There  is  no 
rule  of  thumb  for  run  length  or  number  of  replications,  each  is  model 
dependent.  The  number  of  replications  affects  statistical  accuracy  of 
performance  measures;  specifically,  it  affects  the  width  of  any 
confidence  interval  estimators  (Carson,  2003:  22). 


Design  of  Experiment  (DOE)  itself  plays  a  unique  role  in  designing  and 
conducting  a  simulation  experiment.  DOE  is  the  art  of  building  a  product  or  process  by 
having  all  the  factors  under  control.  It  reduces  time  to  design  and  develop  new  products 
and  processes,  improves  performance  of  an  existing  system,  and  achieves  product  and 
process  robustness.  In  the  context  of  a  simulation  experiment,  it  helps  us  obtain  more 
and  better  information  about  system  performance  using  less  resources.  For  most 
simulation  studies,  it  is  impossible  to  analyze  all  possible  combinations  of  factors 
involved  with  the  model  and  how  they  affect  measures  of  system  performance  and 
effectiveness.  At  this  point,  DOE  is  used  as  a  solution  technique  to  decide  which  factor 
combinations  or  design  points  to  include  in  the  experiment.  Many  software  packages 
include  DOE  tools  such  as  the  Process  Analyzer  embedded  in  Arena,  Design-Expert™ 
PQ  systems™  and  JMP. 

Document  and  Present  the  Simulation  Results: 

The  basic  aim  of  simulation  modeling  is  to  give  accurate  insight 
to  a  decision  maker  about  a  real  system,  not  to  make  a  decision.  In 
general  modelers  spend  most  of  their  time  on  understanding  a  problem, 
building  a  conceptual  model,  implementing  verification  and  validation 
and  obtaining  outputs  from  a  model,  leaving  little  time  to  work  on 
packaging  the  results.  If  the  results  are  not  clearly,  concisely  and 
convincingly  presented,  they  will  not  be  used.  The  presentation  of  the 
results  of  the  study  is  a  critical  and  important  part  of  the  study  and 
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must  be  as  carefully  planned  as  any  other  part  of  the  project 
(Sadowski,  1993:  65-68). 

2.4.  Arena 

There  are  many  simulation  software  packages  available  if  a  modeler  prefers  not  to 
write  his  own  computer  codes.  This  preference  may  be  originated  due  to  limited  time. 
Arena  is  one  of  the  simulation  modeling  tools  which  is  flexible  and  powerful  to  represent 
real  systems.  It  was  released  by  Systems  Modeling  Corporation  in  1993.  It  has  been  used 
as  a  simulation  modeling  tool  in  manufacturing  areas,  health  care  systems,  call  centers, 
warehousing,  transportation  systems,  and  so  on. 

“Simulation  analysts  place  graphical  objects  -  called  modules  -  on  a  layout  in 
order  to  define  system  components  such  as  machines,  operators,  and  material  handling 
devices.  Arena  is  built  on  the  SIMAN  simulation  language.  After  creating  a  simulation 
model  graphically,  Arena  automatically  generates  the  underlying  SIMAN  model  used  to 
perform  simulation  runs”  (Markovitch  and  Profozich,  1996:  437).  Arena  consists  of 
several  templates  which  can  be  defined  as  collections  of  more  than  one  module. 

Beside  these  templates,  Arena  has  useful  animations  and  graphical  tools.  These 
features  visually  provide  clear  understanding  about  processes  in  the  model.  Model 
verification  and  validation  can  be  aided  through  the  use  of  these  features. 

Arena  itself  is  an  effective  tool  which  combines  a  number  of  analysis  tools  in  one 
package.  There  are  three  additional  tools  in  Arena,  including  the  Input  Analyzer,  the 
Output  analyzer  and  the  Process  analyzer.  The  Input  Analyzer  is  an  effective  tool  to 
analyze  raw  input  data  and  determine  a  best  fit  distribution  for  incorporation  in  an  Arena 
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model.  The  Output  Analyzer  is  used  after  running  a  simulation  model  and  includes 
various  ways  to  display  output  data  graphically.  The  Output  Analyzer  also  provides 
confidence  intervals,  one-way  analysis  of  variance  and  comparisons  of  alternative 
systems.  Lastly,  the  Process  Analyzer  is  used  as  an  experimental  design  tool  to  see  the 
system’s  response  to  different  controls  defined  by  the  analyst.  These  controls  can  be 
possible  number  of  cashiers,  servicemen  in  a  gas  station,  repairing  stations,  and  so  on.  By 
using  the  Process  Analyzer,  the  affects  of  different  system  policies  can  be  evaluated 
without  modifying  the  original  model. 

2.5.  Previous  research 

Previous  related  research  analyzing  the  Oncel  training  squadron  as  well  as 
simulation  studies  of  similar  systems  is  discussed  in  this  section.  Cpt.  Davut  Aslan’s 
(2003)  research  focused  on  the  scheduling  problem  of  the  Oncel  squadron  daily  sortie 
program.  In  his  thesis,  the  time  consuming  scheduling  problem  of  planning  the  daily  and 
weekly  flight  program  is  thoroughly  examined.  His  problem  definition  included  certain 
basic  resource  constraints  including  number  of  instructors,  bandits  and  aircraft.  In 
addition  his  efforts  showed  that  the  effects  of  runway  availability,  non-flight  duties  and 
personnel  activities  can  not  be  ignored.  Aslan  (2003)  used  integer  programming  and 
scheduling  techniques  to  approach  the  problem.  The  objective  function  and  constraints 
were  formulated  by  considering  all  those  factors  stated  above.  His  thesis  also  emphasized 
that  the  prediction  of  the  end  of  the  training  term  is  possible  by  modifying  the  time 
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parameters  in  his  integer  formulations  from  weekly  to  new  time  parameters  including  the 
whole  training  program  (Aslan,  2003). 


For  another  effort,  Maj.  June  Rodriguez  (2007)  was  tasked  to  assess  whether  the 
USAF  current  cargo  aircraft  flying  training  resource  capacity  can  accommodate 
additional  student  pilot  demand  due  to  an  increase  in  C- 17s  and  the  newly  implemented 
C-17  Abeam  tactical  requirement.  She  approached  the  problem  by  constructing  a 
simulation  to  analyze  the  flight  training  term  using  Arena.  Her  simulation  models  aircraft 
scheduling  of  various  combinations  of  C-5s,  KC-135s  and  C-17s  from  FY07  to  FY1 1. 
“Current  syllabi  for  all  three  platforms,  with  consideration  to  future  training  starting  in 
FY07,  were  used  to  model  sortie  profiles.  The  model's  findings  include  the  results  of 
1 ,000  runs  for  each  F  Y.  Multiple  replications  reduce  the  impact  of  modeling  anomalies 
and  show  the  overall  limits  on  the  resources”  (Rodriguez,  2007). 

Maj.  Rodriguez’s  model  computes  the  total  time  in  training  days  (noted  in  the 
model  as  time  in  system  (TiS)  each  pilot  needed  to  complete  the  flying  training  portion  of 
his  or  her  curriculum.  The  model  output  also  includes  the  number  of  pilot  types 
graduating  from  each  course.  “The  Graduate  Program  Requirements  Document  (GPRD) 
is  compared  to  the  pilot  graduates  from  the  model  while  the  model  TiS  is  compared  to  the 
class  type  allotted  flying  training  days”  (Rodriguez,  2007). 


The  simulation  is  verified  by  tracking  individual  entities  through 
several  key  points  in  the  system.  The  animation  option  in  ARENA 
facilitates  the  verification  process  by  allowing  visual  observation  of 
proper  model  behavior.  All  assumptions  are  tested  to  verify  proper 
coding  in  the  model.  Two  methods  of  validation  are  performed  for 
the  flying  training  model.  The  projected  flying  hours  for  the  flying 
training  is  compared  to  the  allotted  TiS  in  the  Programmed  Flying 
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Training  (PFT)  plan.  This  comparison  shows  the  TiS  from  the 
simulation  is  comparable  to  the  PFT  and  remains  a  valid 
representation  of  reality.  The  second  method  of  validation  is 
conducted  by  several  pilot  instructors  as  the  subject  matter  experts 
(SMEs)  (Rodriguez,  2007). 

Faas  (2003)  conducted  research  about  an  Air  Force  sortie  generation  process  by 
applying  simulation  model  technique.  In  his  thesis,  it  states  that  Air  Force  decision 
makers  need  a  simulation  tool  to  study  the  effects  of  the  emerging  Autonomic  Logistic 
System  (ALS)  technologies  on  the  Air  Force  sortie  generation  process.  ALS  is  a 
proactive  approach  to  logistics  operations  instead  of  current  reactive  approach.  To 
analyze  two  systems,  the  discrete  event  simulation  model,  focusing  on  the  F- 16  aircraft 
and  the  radar  subsystem,  is  developed  in  Arena.  In  his  model,  actual  data  from  the  F-16 
radar  systems  are  used.  The  sortie  generation  process’  model  provides  making 
comparisons  between  the  current  maintenance  system  and  the  new  ALS  (Faas,  2003). 
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III.  Methodology 


Introduction 

Chapter  3  focuses  on  how  the  steps  of  constructing  the  simulation  model  were 
applied  to  the  F-16  Pilot  Training  process.  This  chapter  consists  of  three  major  sections; 
first  section  focuses  on  assumptions,  building  the  conceptual  model  and  defining  basic 
processes  in  the  flight  training,  second  one  mentions  improving  the  conceptual  model, 
and  third  section  covers  the  application  of  the  seven  steps  of  simulation  modeling  to  the 
model  built  in  Arena. 

3.1.  Assumptions 

The  training  period  in  143rd  Oncel  Squadron  basically  consists  of  two  main 
training  terms  including  ground  training  and  flight  training.  Ground  training  includes  the 
pre  flight  theoretical  lessons,  physical  training  and  simulator  training.  In  addition  to  the 
pre  flight  theoretical  lessons,  there  are  certain  lectures  which  are  planned  before  or  during 
every  sub  phase  of  the  flight  training  (INT,  ACM  and  BFM,  etc.).  These  lectures  are  also 
regarded  as  ground  training.  Scheduling  these  lectures  in  a  timely  manner  is  an  important 
issue  to  prepare  SPs  for  the  upcoming  flight  phase  and  to  make  the  whole  course  more 
effective.  However,  the  ground  training  period  is  not  explicitly  considered  in  this  study 
due  to  the  fact  that  SPs  are  generally  scheduled  for  these  lectures  at  the  end  of  flight 
blocks  or  between  two  flight  blocks,  sometimes  even  when  flights  are  cancelled  due  to 
weather  conditions.  Another  reason  not  to  consider  pre  flight  lessons  and  physical 
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training  in  the  simulation  model  is  that  these  training  phases  are  not  scheduled 
individually. 

Simulator  training  consists  of  15  missions.  Nine  of  these  simulator  missions, 
which  are  supposed  to  be  accomplished  before  the  actual  flight  training  period,  are  not 
taken  into  consideration  in  this  study.  The  remaining  simulator  missions,  which  are 
scheduled  as  the  actual  flight  training  continues,  are  embedded  into  the  model  since  they 
significantly  affect  the  scheduling  process  as  a  major  constraint.  As  a  result,  flight 
training  phase  and  six  simulator  missions  form  the  main  focus  of  this  research. 

Maintenance  activities  are  not  modeled  as  a  separate  sub  model  in  this  study.  It  is 
assumed  that  maintenance  interactions  are  reflected  in  every  sortie  logic  patterns  through 
abort  rate  and  refly  rate.  Abort  rate  covers  all  maintenance  problems  which  may  happen 
between  taxi  and  take  off.  Also  refly  rate  is  considered  covering  all  instances  and 
possibilities  from  SP’s  unsatisfactory  performances  to  maintenance  problems  after  take 
off.  In  addition  to  ideas  above,  spare  aircraft  policies  are  regarded  as  a  solution  to  the 
maintenance  problems  before  taxi.  A  pilot  may  change  his/her  aircraft  in  case  of 
maintenance  problem  and  continue  his/her  mission  as  planned. 

3.2.  Description  of  Flight  Training  in  light  of  Simulation  Model  Concepts 

The  F-16  Pilot  Training  under  the  Oncel  squadron  structure  lasts  approximately 
140  weekdays  (7  months),  35  weekdays  of  this  period  are  planned  as  a  pre-flight  ground 
training,  and  105  weekdays  as  a  flight  training.  As  stated  early  in  Chapter  2,  this 
research’s  objective  is  to  focus  only  on  the  flight  training  part,  defined  as  a  period 
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between  ground  training  and  graduation  as  shown  at  Figure  3.  By  considering  these 
assumptions,  model’s  initial  time  is  assumed  as  a  time  when  the  first  SP  enters  into  flight 
training  just  after  finishing  the  ground  training. 


F-16  PILOT  TRAINING  FLOW  CHART 


Figure  3.  The  Flow  Chart  of  F-16  Pilot  Training  in  Oncel  Squadron 


In  a  year,  Oncel  squadron  consecutively  plans  to  provide  two  separate  flight 
trainings  for  two  separate  SP  groups.  There  is  an  overlap  between  these  two  SP  groups 
because  each  training  period  lasts  approximately  seven  months.  This  overlap  shows  that 
two  different  groups  are  in  the  squadron  at  the  same  time  approximately  during  the  first 
and  the  last  approximately  one  month  of  each  training  period.  The  following  groups  deal 
with  ground  training  during  overlapping  time  periods.  Usually  one  term  starts  flight 
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training  in  the  middle  of  February,  the  other  starts  in  the  middle  of  July.  This  circulation 
continues  without  any  pause  as  one  SP  group  starts  flight  training  whenever  the  previous 
group  completes  flight  training.  These  initial  dates  and  training  periods  may  change 
depending  on  several  factors,  like  number  of  SPs,  IPs,  Bandits,  aircraft  and  weather 
forecast. 

The  F-16  Pilot  Flight  Training  period  consists  of  three  basic  phases  (BTR,  AA, 
and  AG)  which  are  divided  into  sub  phases  as  well.  The  overall  flight  training  process, 
phases  and  sub  phases  are  conceptually  described  in  Figure  3.  The  phases’  sequence,  as 
shown  at  Figure  3,  provides  gradual  progress  in  SP’s  proficiency  as  he/she  accomplishes 
missions  which  vary  from  basic  aircraft  handling  missions  to  more  complex  missions.  As 
stated  in  Oncel  squadron  syllabus,  the  first  objective  of  all  activities  in  the  squadron  is  to 
train  student  pilots  who  fly  an  F-16  at  its  full  performance  with  excellent  knowledge  and 
flight  safety.  To  reach  this  objective,  every  SP  is  normally  scheduled  to  fly  62  sorties 
within  flight  training.  Each  phase  and  sub  phase  of  flight  training  has  a  certain  number  of 
flights  stated  in  the  syllabus.  Sortie  quantities  at  each  phase  and  sub  phase  are  shown  in 
Table  3. 


Table  3.  Sortie  Numbers  in  each  phase  and  sub  phase 


BTR 

AIR  TO  AIR 

AIR  TO  GROUND 

NIGHT 

EVALUATION 

INT 

BFM 

ACM 

SA 

SAT 

IQ 

TQ 

TOTAL 

9 

7 

13 

9 

8 

11 

3 

1 

1 

62 

In  every  phase  and  sub  phase,  each  sortie  can  be  regarded  as  a  prerequisite  for  the 
next  sortie  except  for  night  phase  sorties.  Night  training  sorties  can  be  scheduled  along 
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with  the  AG  phase.  Eventually  an  SP  cannot  be  scheduled  for  a  sortie  without 
accomplishing  the  previous  one.  Besides  actual  sorties,  each  SP  is  required  to  accomplish 
15  simulator  flights,  the  first  six  which  are  scheduled  before  flight  training  phase  as 
mentioned  in  the  previous  section.  Then  the  last  nine  simulator  missions  are  scheduled 
between  actual  sorties  as  proposed  in  the  syllabus.  Table  4  shows  the  simulator  mission 
number  and  the  sortie  number  at  which  a  particular  simulator  mission  is  required  to  be 
completed. 


Table  4.  Simulator  missions 


Simulator  Training 

No 

Time 

Prerequisite 

SIM-7 

1.20 

TR-8 

SIM-8 

1.20 

INT-1 

SIM-9 

1.20 

INT-1 

SIM- 10 

1.20 

INT-1 

SIM- 11 

1.20 

INT-5 

SIM- 12 

1.20 

INT-5 

SIM- 13 

1.20 

SA-1 

SIM- 14 

1.20 

SA-2 

SIM- 15 

1.20 

SAT-1 

While  flight  training  period  continues,  SPs  are  required  to  complete  nine 
simulator  missions  as  shown  at  Table  4.  At  this  point,  another  important  scheduling 
factor  arises  for  simulator  flights  as  well  as  sortie  scheduling.  As  a  squadron  training  rule, 
an  SP  can  not  be  scheduled  for  a  simulator  mission  without  accomplishing  the  previous 


27 


one  which  is  regarded  as  a  prerequisite.  In  the  syllabus,  each  of  nine  simulator  missions  is 
stated  as  a  prerequisite  for  a  certain  sortie,  thus  each  mission  is  required  to  be 
accomplished  before  pre-stated  sortie.  As  seen  from  Figure  3,  BTR,  AA,  and  AG  phases 
have  a  certain  number  of  simulator  missions  which  need  to  be  scheduled  before  certain 
sorties.  During  these  phases,  simulator  missions  should  be  scheduled  consecutively  as 
well  as  actual  flights.  However,  scheduling  constraints  for  simulator  missions  are  not  as 
strict  as  the  ones  for  actual  sorties.  Simulator  missions  are  not  supposed  to  be  scheduled 
immediately  before  a  sortie  whose  syllabus  events  dictate  that  certain  simulator  mission 
as  a  prerequisite.  Instead,  they  can  be  scheduled  at  any  other  feasible  time  before  the 
appropriate  sortie.  From  the  scheduler’s  perspective,  simulator  mission  scheduling  is 
regarded  more  flexible  than  actual  flight  scheduling.  In  most  cases,  SPs  are  first 
scheduled  for  the  sorties  on  the  following  day’s  flight  program,  and  then  any  SPs,  who 
are  not  assigned  to  fly,  are  considered  for  simulator  flights. 

Before  describing  the  conceptual  model,  it  is  important  to  mention  the  practical 
daily  objective  of  the  143rd  Oncel  squadron.  Figure  4  shows  that  Oncel  squadron  is 
responsible  for  producing  two  schedules  for  each  following  day  by  considering  available 
resources,  including  IP,  Bandit,  aircraft,  runway  and  simulator.  Before  squadron  staff 
leave  the  squadron,  the  flight  and  simulator  mission  schedules  are  prepared  and  published 
everyday  by  the  scheduler  (IP  or  Bandit)  who  is  assigned  to  the  scheduling  section  of  the 
Oncel  squadron. 
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Figure  4.  System’s  Inputs  and  Outputs 


3.3.  Conceptual  Model  through  Full  Model 

A  conceptual  model  needs  to  consider  every  process  and  interaction  of  the  system 
under  study.  The  conceptual  model  for  the  F-16  Pilot  Training  was  built  after  reviewing 
the  squadron’s  syllabus  and  gathering  sufficient  information  covering  the  entire  training 
period.  At  first,  the  conceptual  model  is  structured  simply  as  recommended  in  many 
simulation  articles.  Then  the  model  is  developed  in  more  detail  to  capture  all  steps  that 
are  considered  crucial  and  critical  factors  in  the  system. 

Before  stepping  forward  to  build  the  conceptual  model,  the  model’s  key  factors, 
consisting  of  entity  and  resources,  should  be  determined  precisely.  Specifying  the  entity 
type  gives  an  answer  to  what  flows  and  interacts  in  the  model.  In  the  Flight  Training’s 
conceptual  model,  a  certain  number  of  Student  Pilots  are  created  at  once  by  the  Create 
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module  (in  Arena)  and  disposed  as  a  graduated  pilot.  On  the  other  hand,  IPs,  Bandits  and 
Aircraft  are  considered  as  model’s  main  resources  as  well  as  the  runway  and  simulator. 
Resource  quantities  are  shown  in  Table  5. 


Table  5.  Numbers  of  resources 


IP 

BANDIT 

F-16C 

F-16D 

RUNWAY 

SIMULATOR 

20 

10 

20 

8 

1 

1 

Additional  details  are  required  for  our  IP  and  Bandit  resources.  IP  is  a  pilot  who  is 
authorized  to  fly  an  F-16D  (double  seat  model)  with  an  assigned  SP  within  the  training 
period.  A  Bandit  is  a  pilot  who  is  authorized  to  fly  supportive  missions  within  missions 
scheduled  for  SPs.  A  Bandit  is  not  allowed  to  be  scheduled  with  an  SP  for  sorties  which 
require  F-16D  aircraft.  In  the  Oncel  squadron  syllabus,  all  requirements,  including 
number  of  aircraft  to  prerequisites,  are  stated  precisely  for  every  mission. 

After  specifying  basic  units  in  the  simulation  model,  all  processes  and 
interactions,  which  may  take  place  during  a  usual  training  day,  should  be  considered 
thoroughly  and  reflected  sufficiently  into  the  simulation  model.  What  an  SP  does  in  the 
squadron  during  a  usual  day  highlights  the  basic  structure  of  the  model.  If  an  SP  is 
scheduled  for  a  sortie,  he/she  is  supposed  to  meet  all  the  requirements  the  day  before  as 
listed  in  the  syllabus.  In  the  flight  day’s  morning  following  the  mass  briefing,  an  SP  joins 
a  pre-flight  briefing  with  his/her  assigned  IP  two  hours  before  take  off.  They  step  from 
the  squadron  at  least  30  minutes  before  take-off  time.  After  flight,  the  IP  and  SP  debrief 
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and  critique  the  sortie.  These  phases,  regarded  as  sub  processes  of  a  mission,  are  included 
in  the  definition  of  a  sortie.  Among  the  Air  Force  community,  “A  sortie  starts  with  pre¬ 
briefing  and  ends  with  debriefing”  is  a  well-known  quote  concerning  sortie  definition.  In 
the  conceptual  model,  a  sortie  is  defined  as  the  entire  process  covering  pre-briefing,  flight 
and  debriefing.  One  sortie  lasts  approximately  4  hours  which  includes  1  hour  pre-flight 
briefing,  .5  hour  preparation,  1.5  hour  flight,  and  1  hour  debriefing. 

3.3.1.  Basic  Sortie  Flow  in  Transition  Phase 

TRANSITION  PHASE  SORTIE  FLOW 


RESOURCES 

IP,  BANDIT  AND  F-16C/D 


Figure  5.  2-ship  Sortie  in  the  Conceptual  Model 

A  basic  sortie  is  conceptually  designed  as  shown  in  Figure  5.  This  sortie  flow 
diagram  changes  slightly  because  sortie  content  differs  according  to  each  training  phase. 
These  changes  will  be  stated  in  the  following  sections.  Conceptually  this  sortie  flow 
diagram  can  be  applied  for  2-ship  missions  in  every  phase. 
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A  daily  flight  schedule  and  simulator  schedule  are  prepared  primarily  by 
considering  resource  constraints,  SPs’  needs,  and  training  effectiveness.  In  a  daily 
schedule,  available  resources,  consisting  of  IP,  Bandit,  aircraft  and  simulator  are 
allocated  as  effectively  as  possible  to  each  SP  according  to  his/her  need,  sortie  level  and 
current  status  of  the  entire  training.  For  each  SP,  resource  allocation  is  done  by  adding  a 
Seize  module  and  specifying  resource  requirements  stated  in  the  syllabus  for  every  sortie 
as  shown  in  Figure  6. 


Figure  6.  Resource  allocation  for  2-ship  sortie 


As  a  training  principle,  an  SP  scheduled  for  a  sortie,  is  responsible  to  prepare  all 
required  items  in  the  syllabus  before  the  pre-flight  briefing.  Regardless  of  the  weather 
condition,  IP  and  SP  get  together  for  pre-flight  briefing  and  all  syllabus  items  are 
mentioned  and  discussed  thoroughly.  At  the  end  of  the  pre-flight  briefing,  current  and 
future  forecast  reports,  which  are  available  at  the  flight  desk,  are  checked  carefully  before 
stepping  out  from  the  squadron  Although  the  weather  factor  is  mentioned  in  the 
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following  sections,  it  may  be  appropriate  at  least  to  give  a  short  description  here.  Weather 
factor  is  inserted  into  model  as  a  separate  sub  model  which  provides  monthly  rates  based 
on  the  days  when  sorties  are  accomplished  over  two  years. 

If  the  flights  are  not  cancelled,  IP  and  SP  step  out  at  the  specific  time  depending 
on  current  flight  training  phase  and  take-off  following  ground  procedures.  As  shown  in 
Figure  5,  abort  logic  comes  after  weather  and  seize  module  as  a  second  decision  factor. 
Abort  is  defined  as  an  action  to  cancel  the  sortie  due  to  a  maintenance  problem  during 
taxi  or  take-off.  From  the  flight  safety  and  squadron  rules’  perspective,  it  is  not 
recommended  to  get  a  spare  aircraft  due  to  failure  after  starting  taxi. 

Following  the  flight  and  landing,  SP  and  IP  debrief  and  critique  the  mission  in 
order  to  analyze  the  positive  and/or  negative  events  concerning  the  sortie.  Process 
module  is  used  as  a  delay  to  simulate  entire  sortie  duration  including  pre-flight  briefing, 
flight  period,  and  debriefing  which  is  assumed  to  be  distributed  uniformly  between  3.75 
and  4.25  hours.  A  sortie  is  modeled  in  this  fashion  because  the  sub  events  are  not  points 
of  interest  for  this  study. 

At  the  end  of  the  debriefing,  IP  may  decide  that  SP  should  be  rescheduled  for  that 
sortie  due  to  his/her  performance  or  bad  weather  condition,  etc.  In  the  Oncel  training 
syllabus,  it  is  stated  the  squadron  re-flies  eight  percent  of  its  sorties  throughout  each 
training  term.  This  rate  is  reflected  in  the  model  by  adding  a  Decide  module  after  the 
Sortie  Process  module.  In  Figure  5,  a  Delay  module  is  used  to  prevent  SPs  from  flying 
more  than  one  sortie  in  a  day.  As  a  squadron  rule,  SPs  are  scheduled  for  just  one  sortie  in 
a  day,  except  for  some  sorties  and  special  conditions.  At  the  end  of  basic  sortie  flow, 
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resources  seized  at  the  beginning  of  sortie  are  released  by  adding  a  Release  module 
(TUAF  SOP). 

3.3.2.  Basic  Sortie  Flow  with  4-ship  Flight 

Certain  sorties,  especially  after  transition  flight  phase,  require  4-ship  flight 
scheduling  as  indicated  in  the  squadron  syllabus.  These  4-ship  missions  are  reflected  in 
the  conceptual  model  as  shown  at  Figure  7. 

SAMPLE  4-SHIP  SORTIE  FLOW 


RESOURCES 

IP, 

BANDIT 

F-16D&C 


NORMALLY  TWO  SPs  ARE  GROUPED  TO  FLY  IN  4  SHIP  FLIGHT.. 


Figure  7.  4-ship  Sortie  in  the  Conceptual  Model 

In  flight  training,  there  are  two  types  of  4-ship  sorties.  The  first  one  requires  4- 
ship  scheduling  specifically  during  2vs2  intercept  phase.  The  second  one  encourages  4- 
ship  scheduling,  however  it  has  flexibility  to  select  between  2-ship  and  4-ship  scheduling 
based  on  number  of  SPs  waiting  for  the  same  sortie  or  resource  levels  (specifically  during 
SA  and  SAT  phases).  This  flexibility,  which  prevents  unnecessary  waiting  time  if  an  SP 
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is  behind  in  their  training  or  there  is  no  other  SP  to  be  batched,  is  reflected  in  4-ship 
sortie  logic  except  for  the  Intercept  phase. 

Resources  seized  by  SPs  during  4-ship  sorties  are  also  considerations.  In  the 
syllabus,  the  requirements  and  prerequisites  for  each  sortie  are  stated  in  detail  to  include 
how  many  F-16  C/D  model  aircraft,  SPs,  IPs  and  Bandits  should  be  scheduled.  Generally 
SPs  who  are  at  the  same  sortie  level,  are  scheduled  in  doubles  for  the  4-ship  required 
mission  throughout  the  Intercept,  SA  and  SAT  sub  phases.  The  application  of  this  rule 
provides  significant  savings  in  terms  of  resources  and  effective  allocation.  For  example, 
INT-5  sortie,  a  4-ship  mission,  requires  3  bandits  as  support  pilots,  1  IP,  3  F-16  C  and  1 
F-16  D  aircraft  as  shown  at  Figure  8.  These  resources  are  allocated  for  just  one  SP  for 
one  sortie.  By  scheduling  two  SPs  concurrently,  two  training  sorties  can  be  accomplished 
using  two  Bandits,  two  IPs,  two  F-16  C  and  two  F-16  D  aircraft  for  two  SPs.  As  a  result, 
the  scheduling  method  for  4-ship  missions  encourages  batching  two  SPs  who  are  at  the 
same  training  level.  The  rest  of  the  4-ship  sortie  has  the  same  flow  as  the  basic  transition 
sortie  flow  shown  at  Figure  5. 


Figure  8.  Allocations  for  4-ship  Sorties 
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3.3.3.  Main  Model  and  Sortie  Logic 

The  main  model  which  reflects  62  sorties  and  interactions  between  SPs  and 
resources  (IPs,  Bandits  and  F-16  C/D)  in  Arena  is  shown  at  Figure  9.  The  main  flow 
consists  of  sub  models  which  correspond  to  the  sub  phases  of  the  flight  training  program. 
The  simulation  model  starts  by  creating  a  certain  number  of  SPs  and  terminates  when  the 
last  SP  flies  the  last  sortie  of  the  entire  training  program. 
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Figure  9.  Main  Model  in  Full  Model 

3.3.3. 1.  2-ship  Sortie  in  Full  Model 

At  the  end  of  several  modifications,  2-ship  sortie  logic  is  built  as  shown  at  Figure 
10.  The  first  part  belongs  to  the  simulator  mission,  and  the  rest  describes  the  steps  and 
logic  for  the  2-ship  sortie. 
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3.3.3.2.  4-ship  Sortie  in  Full  Model 

The  4-ship  sortie  logic  was  finalized  following  several  reviews  and  verifications 
to  meet  the  actual  constraints  for  4-ship  sorties  in  the  system  as  shown  at  Figure  11.  The 
first  is  for  simulator  missions;  the  rest  covers  all  possible  situations  which  may  occur 
within  4-ship  sorties. 
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Figure  11.  4-ship  Sortie  in  Full  Model 
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3.4.  Auxiliary  Process  Models 

The  flight  training  process  is  regarded  as  the  main  process  within  the  model. 

There  are  some  sub  processes  which  contribute  significant  inputs  and  effects  to  the  main 
process  and  are  included  as  sub  models. 

3.4.1.  Duties  during  Training  Period 

In  the  Section  3.2,  system  resources  are  stated  clearly.  These  resources,  like  IPs 
and  Bandits,  are  used  mainly  for  SPs’  training  purposes.  Besides  allocating  these 
resources  to  the  SPs,  there  are  three  additional  duties  that  require  use  of  these  resources 
over  the  entire  training  period.  These  duties  consist  of  Supervisor  of  Flight  (SOF), 
Runway  Supervisor  Unit  (RSU),  and  Range  Training  Officer  (RTO).  The  scheduler  must 
take  into  consideration  assigning  an  IP  or  Bandit  to  these  duties  each  day  while  allocating 
resources.  SOF  and  RSU  duties  require  authorized  personnel  (IP  or  Bandit)  as  long  as 
daily  flights  continue  over  all  phases.  Additionally,  RTO  is  a  duty  which  requires 
authorized  personnel  (IP  or  Bandit)  as  long  as  the  range  area  is  in  use.  The  range  area  is 
in  use  during  the  Air  to  Ground  phase  which  consists  of  SA  and  SAT  sub  phases.  Pilots, 
assigned  to  one  of  the  duties,  will  be  classified  as  unavailable  in  terms  of  flight 
scheduling  in  that  sortie  block. 

These  additional  duties  are  implemented  in  the  simulation  model  by  adding  a 
separate  Create  Module  for  each  duty  type.  Entities  are  defined  as  duties  which  are 
created  periodically  and  follow  logic  to  check  the  duty  schedule,  flight  schedule  and 
flight  cancellation,  for  every  predetermined  period.  This  logic  prevents  assigning 
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resources  (IP  and/or  Bandit)  while  there  is  no  flight  activity.  The  sub  models  are  shown 
in  Figures  12  through  14  for  all  three  duties.  Since  143rd  Oncel  squadron  is  not  the  only 
squadron  stationed  at  Akinci  AFB,  these  three  duties  are  allocated  equally  between 
squadrons  by  considering  the  number  of  flight  blocs  (Kayhan,  2007).  Although  there  are 
some  rules  in  allocating  the  duties  between  squadrons,  any  squadron  may  request 
immediate  change  in  scheduling  due  to  sudden  personnel  constraints.  By  considering  all 
these  ideas,  the  overall  duty  rate  of  the  Oncel  squadron  is  taken  and  inserted  into  the 
model. 


Figure  12.  Design  in  Arena  for  RSU 


Figure  14.  Design  in  Arena  for  RTO 
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3.4.2.  Annual  Leave  for  IPs  and  Bandits 


Another  significant  factor  which  should  be  considered  in  the  model  is  the  number 
of  personnel  on  leave  during  the  training  period.  Normally  IPs  and  Bandits  are  the  only 
personnel  whose  leave  status  may  affect  the  system,  since  SPs  do  not  have  the  ability  to 
take  leave  except  under  special  conditions  which  are  not  considered  within  the  simulation 
model. 

A  person  is  allowed  to  take  as  much  as  1  month  of  leave  in  a  year.  Although  there 
is  no  strict  rule  in  scheduling  off  days,  it  is  preferred  that  personnel  take  smaller  periods 
of  leave,  like  ten  days  in  winter  and  the  rest  in  summer.  These  days  include  weekends,  so 
there  are  8  weekdays  off  for  winter  term  and  14  weekdays  for  summer  term.  Since  the 
simulation  runs  without  considering  weekends  and  schedules  are  determined  according  to 
weekdays,  8  and  14  weekdays  are  used  in  the  Leave  sub  model. 

It  is  obvious  that  the  number  of  the  personnel  on  leave  affects  how  many  daily 
sorties  can  be  flown.  To  minimize  these  effects,  there  are  some  regulations  to  follow 
regarding  leave.  The  most  important  rule  affecting  our  model  limits  the  number  of 
personnel  on  leave  to  25  percent  of  total  personnel.  Consequently,  the  number  of 
personnel  on  leave  is  not  allowed  to  exceed  25  percent  of  all  personnel  (IPs  and  Bandits). 

Two  different  Leave  sub  models,  one  for  IPs  and  one  for  Bandits,  are  added  into 
the  simulation  model  with  similar  logic  because  IPs  and  Bandits  have  different 
responsibilities  in  the  squadron.  The  limitations  mentioned  above  are  reflected  in  the 
simulation  model  by  various  formulations  and  modules.  Time  periods  for  winter  and 
summer  are  defined  and  a  certain  number  of  entities,  equal  to  the  total  number  of  IPs  and 
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Bandits,  are  created  once  and  circulated  between  winter  and  summer  leave  processes 
until  meeting  predefined  conditions  and  then  disposed  after  using  total  amount  of  leave 
days.  As  an  initial  condition,  half  of  the  personnel  are  considered  that  they  had  used  their 
winter  leaves  during  the  previous  year’s  winter  period  because  the  simulation  model  is 
designed  to  start  running  at  the  middle  of  winter  period.  An  additional  constraint  relating 
to  the  Transition  phase  requires  every  SP  to  fly  most  of  their  sorties  with  the  same  IP.  So 
IPs  are  not  allowed  to  take  leave  during  the  Transition  phase.  On  the  other  hand,  there  is 
no  similar  limitation  for  Bandits. 

In  the  model  after  creating  leave  entities  periodically,  25  percent  rate  limitation  is 
checked  via  the  Decide  module  as  shown  in  Figure  15.  Simulation  time  is  also  checked 
for  IPs  to  be  sure  that  the  Transition  phase  is  completed.  Every  leave  entity  is  allowed  to 
seize  one  of  the  resources  (IP  or  Bandit)  during  certain  periods  (8  or  14  weekdays) 
according  to  the  simulation’s  days.  At  the  end  of  winter  (summer)  leave,  the  entity  enters 
into  a  loop  to  scan  for  necessary  conditions  to  take  summer  (winter)  leave  as  well. 
Finally,  the  leave  entity  is  disposed  of  after  taking  22  weekdays  of  leave. 
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Figure  15.  Sub  model  for  leave  policy  (for  IP  and  Bandit) 
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3.4.3.  Daily  Weather  Condition  Checks 

As  mentioned  early  in  this  chapter,  weather  conditions  are  another  significant 
factor  to  include  in  the  model.  Clearance  to  start  daily  flights  is  strongly  dependent  on  the 
weather  forecast.  Flight  cancellations  due  to  holidays  are  included  in  the  weather  factor. 
Sorties  should  be  accomplished  under  reasonable  weather  conditions  as  much  as  possible, 
especially  during  initial  phases. 


As  shown  at  Figure  16,  the  weather  forecast  is  embedded  into  the  model  by 
adding  a  Create  module  which  creates  ‘daily  forecast  check’  entities  before  the  daily 
flights  start.  The  entity  passes  through  one  of  the  12  Assign  modules  representing  the 
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current  simulation  month.  The  entity  is  assigned  a  monthly  weather  abort  rate  based  on 
proportion  of  weather  cancellations  for  that  month  over  the  past  two  years.  This  rate  is 
defined  as  a  variable  and  used  as  a  decision  factor  to  cancel  the  flights  or  not.  For  cases 
where  the  flights  are  cancelled,  the  runway  is  seized  for  the  whole  day,  allowing  no 
activity. 

3.4.4.  Simulator  Flights 

Simulator  flights  form  one  of  the  important  phases  of  the  entire  training  period  to 
include  the  flight  training  phase.  Within  the  F-16  Pilot  training  course,  there  are  15 
simulator  flights  which  are  distributed  between  actual  sorties  in  order  to  let  the  SPs 
become  familiar  with  procedures  and  responsibilities  on  forthcoming  flight  phases  or 
particular  sorties.  All  simulator  mission  scheduling  constraints  are  precisely  stated  in  the 
syllabus.  The  model  deals  with  the  nine  simulator  schedules  after  flight  training  starts. 
These  nine  simulator  missions  and  sorties  to  which  these  simulator  missions  are  specified 
as  prerequisite  are  listed  in  Table  4  at  the  beginning  of  this  chapter.  Because  there  is  only 
one  simulator  unit  on  base,  one  SP  and  one  IP  or  Bandit  are  scheduled  for  each  simulator 
mission.  In  the  model,  the  simulator  unit’s  schedule  allows  seven  missions  in  a  day.  The 
simulator  unit  and  assigned  pilots,  IP  or  Bandit,  are  considered  as  resources  for  simulator 
missions.  Simulator  mission  model  is  combined  with  the  main  model  by  attaching  it  at 
the  beginning  of  certain  flight  sorties. 
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3.5.  Sub  Model  for  Statistics 


Within  the  simulation  model,  several  statistics  are  collected  and  used  for  various 
intentions.  In  addition  to  statistics  defined  in  Arena  statistic  section,  a  separate  sub  model 
is  built  to  collect  daily  sorties,  abort  and  refly  numbers,  and  this  data  is  used  to  verify 
whether  the  parameters  are  reasonable  or  not.  This  sub  model  is  embedded  into  the 
simulation  model  as  shown  at  Figure  17. 


DAILY  SORTIES 


1 


COLLECTING 
DAILY  SORTIES 


RESETINGfor 
NEXT  DAILY 
SORTIES 


DELAY  for  NEXT 
DAY 


Figure  17.  Daily  sorties  flown 

3.6.  Attributes  and  Variables 

In  this  section  the  attributes  and  variables  used  throughout  the  model  and  in 
collecting  some  statistics  are  listed  and  explained  briefly  in  Table  6.  These  attributes  and 
variables  are  used  as  part  of  the  logic  throughout  the  simulation  model  and  the  flight 
training  process  with  a  sufficient  level  of  detail. 
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Table  6.  Variables  and  attributes  in  the  model 


pass 

Used  as  a  counter  to  count  SPs  during  certain  sortie 

lastSP 

To  specify  the  last  SP 

status  of  SOF  (0  or  1) 

SOF  current  duty  status  and  provide  scheduling  logic 

status  of  RSU  (0  or  1) 

RSU  current  duty  status  and  provide  scheduling  logic 

RTO  STATUS  (0  or  1) 

RTO  current  duty  status  and  provide  scheduling  logic 

total  SP 

quantity  of  total  student 

abort  rate 

value  for  abort  rate 

refly  rate 

value  for  refly  rate 

total  IP 

quantity  of  total  IP 

total  BANDIT 

quantity  of  total  Bandit 

current  leave  summer  BND 

number  of  Bandit  currently  on  leave  during  summer 

current  leave  winter  BND 

number  of  Bandit  currently  on  leave  during  winter 

current  leave  summer  IP 

number  of  IP  currently  on  leave  during  summer 

current  leave  winter  IP 

number  of  IP  currently  on  leave  during  winter 

Weather  (0  or  1) 

monthly  rate  for  flight  according  to  weather  condition 

Cancel  (0  or  1) 

indication  of  whether  flights  are  cancelled  or  not. 

Spnumber 

given  initial  number  of  SP  in  flight  training 

ip  assign 

IP  allocation  during  Transition  phase 

Ipindex 

IP  index  used  in  Transition  phase  to  allocate  IP 

duration 

total  number  of  days  to  complete  the  training  period 

TR(1,..,9) 

counter  for  Transition  sorties 

Intercept  (1,..,7) 

counter  for  Intercept  sorties 

bahcl 

counter  for  aircraft  handling  sortie 

Bfm  (1,..,12) 

counter  for  BFM  sorties 

Acm  (1,..,9) 

counter  for  ACM  sorties 

tmt 

counter  for  TMT  sortie 

Sa  (1,..,7) 

counter  for  SA  sorties 

Sat  (1,..,5) 

counter  for  SAT  sorties 

Sta  (1,..,2) 

counter  for  STA  sorties 

Arec  (1,..,2) 

counter  for  AREC  sorties 

Cas  (1,2) 

counter  for  CAS  sorties 

ntrl 

counter  for  NTR  sortie 

Ni(l,..,2) 

counter  for  NI  1  and  NI  2  sorties 
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3.7.  Verification  and  Validation  of  the  Model 


Verification  is  the  step  to  check  the  model  and  logic  to  ensure  they  are 
implementing  what  is  intended.  Various  verification  techniques  are  mentioned 
sufficiently  in  Chapter  2.  In  this  section,  our  focus  is  on  the  application  of  verification 
methods  for  the  model.  First  it  is  appropriate  to  mention  again  that  verification  should  not 
be  considered  as  a  step  which  is  applied  once  while  building  a  model.  It  is  an  ongoing 
process  where  a  modeler  uses  various  techniques  throughout  construction  of  the  model. 

The  animation  feature  in  Arena  is  one  of  the  major  techniques  used  to  verify  the 
model.  Whenever  additional  logic  or  a  new  sub  model  is  inserted,  the  simulation  is  run 
with  animation  enabled  in  Arena  to  check  for  the  proper  flow  of  entities  and  use  of 
resources.  Arena  has  other  useful  features  such  as  dynamic  variables  to  count  entities  at 
specific  points.  These  counters  are  embedded  into  the  model  to  check  the  results  and 
verify  them  numerically.  For  instance,  a  dynamic  variable  is  inserted  for  every  flight  sub 
phases  to  collect  and  check  the  number  of  simulator  missions  accomplished.  The 
numbers  from  the  simulation  are  compared  with  numbers  obtained  analytically  or  from 
the  actual  system.  This  technique  is  used  in  a  number  of  places  throughout  our  model. 

Another  verification  technique  is  to  have  someone  familiar  with  the  actual  system 
review  the  model.  Main  model  and  sub  models  were  reviewed  by  pilots,  also  current 
AFIT  students,  who  had  attended  flight  training  program  in  143rd  Oncel  squadron  to  see 
whether  the  sortie  flow  logic  2-ship  and  4-ship  is  correctly  represented.  In  addition,  the 
sub  models  representing  the  actual  procedures  for  leave,  weather  and  duty,  were  also 
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reviewed.  Based  upon  feedbacks  from  these  reviews,  the  model  was  modified 
accordingly. 

Historical  data  validation  and  face  validity  are  used  among  the  validation 
techniques.  Data  from  the  last  12  training  terms  was  used  to  validate  the  model.  Values 
from  the  actual  system  and  simulation  model  for  the  entire  training  duration  differed  by 
an  average  of  nine  weekdays  which  is  considered  reasonable.  Also  one  pilot,  who  is  a 
graduate  student  in  the  ENS  department  simulation  track,  reviewed  most  of  the  model’s 
modification  as  a  face  validity  technique. 

3.8.  Conclusion 

This  chapter  focused  on  description  of  the  system,  concepts  of  building  a 
simulation  model  and  application  of  steps  for  building  a  simulation  model.  Details  of  our 
final  model  are  discussed  to  include  numerous  figures  depicting  the  Arena  logic.  In  the 
following  chapter,  simulation  model  analysis  is  focused  on  design  of  experiment, 
regression  and  metamodeling. 
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IV.  Analysis  and  Results 


General 

This  chapter  forms  the  final  steps  of  this  simulation  study.  In  this  chapter  the  main 
focus  is  concentrated  on  design  of  experiment  (DOE)  using  the  final  simulation  model, 
gathering  sufficient  data  from  simulation  runs,  and  making  an  analysis  to  include  various 
statistical  methods.  Previously  in  the  first  chapter,  the  main  concern  of  this  research  is 
explained.  Statistical  analysis,  possible  techniques  and  arguments  are  discussed  in 
Chapter  2  within  the  literature  review.  Actual  construction  of  the  simulation  model  and 
its  main  phases  are  discussed  in  Chapter  3.  Subsequently  the  analysis,  results  and  their 
presentation  are  covered  in  Chapter  4. 

4.1.  Design  of  Experiment  for  Simulation  Model 

As  mentioned  in  the  previous  chapters,  the  143  F-16  Pilot  Training  Squadron 
has  one  performance  measure  and  five  main  factors  as  listed  at  Table  7.  The  main  factors 
consist  of  system  resources  and  entities  (SPs)  which  are  the  key  players  of  the  training 
system.  The  total  time  of  training  periods  is  a  performance  measure  used  as  response  for 
the  system.  In  addition  to  this  performance  measure,  IP  and  Bandit  utilizations  will  be 
used  to  make  inferences  about  some  system  performance. 


50 


Table  7.  Main  Factors 


Main  Factors 
Instructor  Pilot  (IP) 
Bandit 

F-16  C  model 
F-16  D  model 
Student  Pilot  (SP) 


The  first  four  factors  are  the  main  resources  of  the  F-16  Pilot  Training  Squadron 
simulation  model.  SP  is  designed  as  a  main  entity  in  the  simulation  model.  To  figure  out 
which  factor  is  more  significant  for  the  response  (Total  Time)  and  how  they  interact 
between  each  other,  a  2k  factorial  design  is  implemented  by  using  Arena’s  Process 
Analyzer.  In  the  factorial  design,  k  denotes  the  number  of  the  factors.  Each  factor’s  low 
and  high  values  are  specified  by  considering  the  data  belonging  to  previous  training 
terms.  Table  8  shows  the  25  factorial  design. 


Table  8.  25  Factorial  Design 


Factors 

Low  (-1) 

High  (+1) 

IP 

15 

31 

Bandit 

3 

11 

F-16  C 

14 

26 

F-16  D 

6 

14 

SP 

12 

28 
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All  these  design  points,  shown  in  Appendix  B,  are  formed  in  Process  Analyzer, 
and  the  responses  including  the  total  time  of  the  training  period  and  the  utilizations  are 
collected  for  each  design  point.  Each  scenario  is  run  for  30  replications.  This  replication 
number  was  determined  during  the  verification  and  validation  phases  because  it  provides 
enough  runs  to  meet  Central  Limit  Theorem  conditions  for  approximately  normal  output 
data  and  also  provides  us  with  a  reasonable  half  width.  Since  Total  Time  values  are 
expected  higher  than  100  weekdays,  plus  and  minus  four  weekdays  half  width  is 
acceptable  for  our  model.  All  design  point  results  were  checked  for  desired  accuracy  with 
different  replication  numbers  by  using  the  formula  suggested  by  Law  (2007:  501). 

na*  (P)  =  min  {i  >  n:  t,u^  *  (S2(n)/i)1/2  <  p}  (1) 

na*  is  the  number  of  additional  replications  needed  to  achieve  half  width  less  than 
desired  value  of  />'.  S2(n)  denotes  the  variance  with  the  present  replication  number.  Also  i 
denotes  the  iterative  increase  in  the  number  of  replication. 

The  responses  are  collected  after  running  the  simulation  model  for  all  design 
points  in  Process  Analyzer.  The  design  points  and  responses  are  tabled  in  Appendix  C. 
The  data  is  initially  analyzed  by  using  three  different  types  of  plots  including  Interaction 
Plot,  Main  Effects  Plot  and  Cube  Plot  via  Minitab.  Interaction  plots  show  how  two 
factors  affect  each  other  and  the  response  at  low  and  high  values  are  shown  at  figure  18. 
The  line  slope  shows  that  the  equivalent  factor  affects  the  response  (Total  Time) 
depending  on  the  interaction  factor.  If  the  slope  is  steep,  it  means  that  the  factor,  specified 
at  the  bottom  of  the  each  sub  plot,  has  a  significant  effect  on  the  response  based  on  the 
interaction  with  the  matching  factor.  As  seen  at  figure  18,  most  of  the  subplots  show 
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almost  no  gap  between  two  lines  (normal  and  dashed  lines)  corresponding  to  the  low  and 
the  high  values  of  the  factors  specified  at  the  right  side  of  the  figure.  If  there  is  no  gap 
between  the  two  lines,  then  it  can  be  said  that  the  system  shows  no  reaction  to  the  low 
and  high  values  of  matching  factors. 


Interaction  Plot  for  Total  Time 


IP  number 

15 

31 

IP  number 

Bandit 

number 

3 

11 

SP  number 

— *— 

12 

28 

SP  number 

F-16C 

— *— 

14 

26 

F-L6D 


Figure  18.  Interaction  Plot  for  Total  Time 


There  are  three  interaction  subplots,  including  SP-F-16C,  SP-F-16D  and  F-16C-F- 
16D  interactions,  which  show  a  significant  gap  between  the  two  lines.  For  low  SP  value 
(12),  the  F-16C  low  and  high  values  do  not  have  an  effect  on  the  Total  Time.  Contrarily, 
in  case  of  high  SP  value  (28),  both  values  of  F-16C  affect  the  Total  Time;  the  low  F-16C 
value  (14)  results  in  higher  Total  Time  values  than  the  high  F-16C  value.  Same 
discussion  can  be  made  for  the  interaction  between  SP  and  F-16D  factors,  except  the 
dashed  line  corresponding  to  the  high  SP  value  is  less  steep  than  the  one  corresponding  to 
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the  SP-F-16C  interaction.  This  means  that  F-16C  affects  the  Total  Time  more  than  F-16D 


for  the  high  SP  value  case  (28). 

As  it  is  observed,  the  unit  scale  of  each  factor  is  different.  Since  each  line  shows  a 
linear  slope  and  the  slope  determines  how  the  corresponding  factor  affects  the  response 
depending  on  the  matching  factor,  the  unit  scale  does  not  create  a  problem  here.  Lastly  as 
the  interaction  between  F-16C  and  F-16D  is  examined,  it  is  observed  that  there  is  a  gap 
between  two  lines  almost  with  the  same  slope.  For  both  low  and  high  F-16C  values,  the 
low  and  high  F-16D  values  affect  the  Total  Time  with  the  same  rate  (slope).  The  existing 
gap  is  caused  due  to  the  effects  of  the  low  and  high  F-16C  values  over  the  Total  Time 
based  on  the  F-16D  values.  That  is  because  Bandits  can  be  replaced  by  an  IP  and  an  IP 
can  fly  twice  in  a  day.  In  this  case  only  the  utilization  values,  in  other  words  working 
hours,  increase  depending  on  the  other  resources  availability.  Also  it  can  be  inferred  that 
the  system  does  not  need  more  aircraft  for  the  lower  SP  number  cases.  On  the  other  hand, 
aircraft  numbers  become  important  when  the  SP  number  is  high  in  the  system.  Another 
point  is  that  since  the  system  has  substitution  flexibilities  with  the  Pilot  resources,  IP  and 
Bandit.  These  factors  do  not  have  a  significant  effect  on  Total  Time. 

Another  tool  which  provides  visual  explanations  of  the  factor  contribution  to  the 
system  is  main  effects  plot  for  responses.  The  main  effects  plot  interpretation  is  valid 
when  there  is  no  interaction  between  factors.  This  plot  is  used  to  observe  the  changes  in 
the  Total  Time  in  response  to  changing  between  each  factor’s  low  and  high  values.  As  it 
will  be  mentioned  in  the  following  sections,  the  interactions  terms  are  added  into  the 
linear  model  of  the  simulation. 
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Another  concern  in  analyzing  the  main  effects  plot  should  be  on  the  unit  scale  of 
each  factor.  The  intervals  between  low  and  high  values  of  each  factor  are  not  equal  as 
shown  in  Figure  19.  The  interval  for  IP  and  SP  is  16-unit  size,  the  one  for  F-16D  and 
Bandit  is  8-unit  size,  and  the  F-16C  has  12-unit  size  interval.  Since  the  low  and  high 
values  are  used  for  this  analysis,  the  following  comments  are  based  on  the  analysis  by 
just  considering  the  given  numerical  ranges.  Table  9  shows  the  Total  Time  values 
corresponding  to  the  low  and  high  values  of  each  factor  with  the  change  per  unit  by 
considering  Main  Effects  Plot  in  Figure  19. 


Main  Effects  Plot  for  Total  Time 

Data  Means 


Figure  19.  Main  Effects  Plot  for  Total  Time 
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Table  9.  Total  Time  Changes  per  Unit  for  Each  Factor 


Low 

High 

Change  per  unit 
(weekdays) 

IP 

126.759 

122.857 

0.243875 

Bandit 

125.813 

123.803 

0.25125 

SP 

102.385 

147.231 

-2.802875 

F-16C 

135.443 

114.174 

1.772416667 

F-16D 

129.54 

120.076 

1.183 

Each  factor’s  effect  on  a  response  can  be  separately  analyzed  via  main  effect 
plots.  A  steeper  line  indicates  that  factor  has  a  significant  effect  for  a  performance 
measure  (Total  Time).  Apparently  SP  has  the  greatest  effect  on  Total  Time  of  F-16  Pilot 
Training  period.  The  F-16  C  has  the  second  biggest  effect  with  F-16  D  next.  Finally,  IP 
and  Bandit  have  the  least  effect  on  the  response.  F-16  C  contributes  to  the  system  more 
than  F-16  D  because  SPs  are  assigned  for  missions  requiring  F-16  C;  more  than  F-16  Ds. 
Almost  two-thirds  of  all  sorties  require  F-16  Cs. 

As  stated  previously  the  system  is  not  affected  by  the  number  of  IPs  or  Bandits 
significantly  because  IPs  and  Bandits  fly  twice  in  a  day,  but  their  utilization  values 
increase.  The  utilization  values  are  shown  with  Time  values  in  Appendix  C.  Additionally 
Main  Effects  Plots  for  both  IP  and  Bandit  utilizations  are  placed  in  Appendix  D. 

All  factor  combinations’  effects  on  performance  measure  (Total  Time)  are 
displayed  in  Figure  20  called  Cube  Plot.  This  plot  provides  a  summary  of  each  factor 
effect  on  the  response  depending  on  every  combination  of  resources.  Practically,  it  is 
easy  to  see  how  the  particular  factor  (resource)  affects  the  response  depending  on  other 
factors  by  just  observing  these  graphs.  Each  row  consisting  of  two  cubes  separately 
define  the  low  and  high  F-16D  values,  in  the  same  manner,  each  column  of  cubes 
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separately  defines  the  low  and  high  F-16C  values.  Three  edges,  X,  Y  and  Z  axis,  of  the 
cubes  define  the  rest  of  the  factors  respectively  IP,  SP  and  Bandit  values. 
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Figure  20.  Cube  Plot  for  Total  Time 


Cube  plots,  like  main  effect  plots,  are  valid  only  if  there  is  no  interaction  term. 
But  we  used  this  plot  to  reflect  the  DOE  results  visually.  By  just  looking  at  the  corners 
corresponding  to  Bandit  values,  it  can  be  predicted  that  the  Bandit  factor  is  not  a  key 
player  in  the  system  because  it  does  not  show  significant  changes  between  the  low  and 
high  values.  The  same  situation  is  valid  for  the  IP  factor  as  well.  As  mentioned  before, 
IPs  and  Bandits  are  allowed  to  fly  more  than  one  sortie  which  may  result  in  higher 
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utilization  values.  In  addition  to  this,  Bandits  can  be  replaced  by  any  available  IP  in  the 
system.  This  plot  shows  a  parallelism  to  previous  comments. 

4.2.  Metamodeling  and  Regression  Analysis 

A  simulation  model  is  considered  as  a  black  box  with  input  data  inserted  and 
output  data  collected.  An  analyst  should  understand  the  interactions  happening  inside  of 
this  box.  A  simulation  is  a  mathematical  model  of  a  system  that  may  include  many 
detailed  inputs  in  providing  outputs  capturing  system  performance.  A  metamodel  is 
defined  as  a  model  of  a  simulation  model.  In  other  words,  a  new  mathematical  model  is 
built  using  the  output  data  collected  from  a  simulation  model  instead  of  using  real  world 
data  (Miller,  2007).  One  of  the  reasons  leading  to  metamodeling  is  the  high  cost  of 
running  large-scale  simulation  models.  Secondly,  time  savings  can  be  considered  as 
another  reason.  For  our  case,  non  availability  of  the  Arena  software  is  another  factor 
suggesting  use  of  a  metamodel.  We  specifically  form  a  linear  model  using  the  outputs  of 
our  simulation  model. 

The  factorial  design  is  discussed  as  a  source  of  data  and  analysis  tool  to  provide 
insight  about  our  system  in  the  previous  section  of  this  chapter.  The  Multiple  Linear 
Regression  Modeling  is  implemented  by  using  the  data  collected  via  DOE.  The  factorial 
design  and  regression  techniques  go  hand-in-hand  in  building  a  metamodel. 

The  first  step  in  regression  analysis  is  to  specify  the  factors  and  responses.  Our 
factors  (IP,  Bandit,  SP,  F-16C,  and  F-16D)  were  determined  in  Chapter  3  while  building 
our  simulation  model.  The  second  step  is  to  conduct  variable  selection  before  going 
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through  further  analysis.  The  stepwise  technique  for  variable  selection  is  applied  to  select 
which  variables  and  interaction  terms  should  be  included  in  the  regression  analysis  to 
build  a  multiple  linear  model.  This  technique  indicates  which  factors  are  significant  for 
our  flight  training  system.  An  analyst  wants  to  build  a  linear  model  which  provides  the 
same  performance  and  still  meets  the  same  requirements  with  less  number  of  factors  if 
the  model  has  the  same  adequacy. 

Each  factor’s  contribution  to  the  R  square  value  and  the  factor’s  individual  p 
value  are  considered  as  the  evaluation  criteria  during  the  variable  selection.  The  p  value 
was  specified  as  0.05  in  this  evaluation.  The  R  square  value  shows  the  variability  of  the 
system  explained  by  the  linear  model.  The  p  value  shows  whether  the  particular  factor 
contributes  to  model  or  not.  Larger  p  values  denote  less  contribution  to  the  model  and 
these  factors  are  not  included.  At  the  end  of  variable  selection,  IP  number  is  proposed  to 
be  included  in  the  linear  model  since  it  has  acceptable  p  value  although  it  does  not 
contribute  to  R  square  significantly.  Unlike  IP  and  other  factors,  bandit  factor  is 
considered  less  significant  by  considering  its  p  value  shown  at  Table  10.  Although  Bandit 
has  a  higher  p  value,  it  is  added  to  the  model  since  it  changes  the  normality  plot. 

In  addition  to  these  individual  factors,  the  interactions  including  SP-F-16C  and 
SP-  F-16D  are  observed  to  be  significant  and  are  included  in  the  model.  The  interaction 
term  between  IP  and  SP  is  not  included  in  the  model  by  considering  its  contribution  to  R 
square.  The  step  history  for  variable  selection  is  placed  in  Appendix  E. 
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Table  10.  Variable  selection  via  Stepwise  Technique 


SSE 

DFE 

MSE 

RSquare 

RSquare  Adj 

Cp 

390.3 

24 

16.3 

0.984 

0.979 

16.016295 

Lock 

Entered 

Parameter 

Estimate 

nDF 

SS 

"F  Ratio" 

Prob>F 

X 

X 

Intercept 

19.3 

1 

0 

0 

1 

X 

ip 

-0.244 

1 

156.1851 

7.218 

0.0126 

X 

Bandit 

-0.251 

1 

37.6148 

1.794 

0.113 

X 

Sp 

8.01 

3 

18589.11 

286.37 

0 

X 

fl6c 

2.24 

2 

6035.478 

139.46 

0 

X 

fl6d 

1.2 

2 

1245.787 

28.788 

0 

(ip)*  (bandit) 

0 

2 

64.05579 

1.545 

0.2347 

(ip)*(sp) 

0 

1 

94.17409 

5.059 

0.0339 

(ip)*(fl6c) 

0 

1 

40.71178 

1.953 

0.175 

(ip)*(fl6d) 

0 

1 

18.16236 

0.834 

0.3703 

(bandit)*(sp) 

0 

2 

51.79528 

1.218 

0.3143 

(bandit)*(fl6c) 

0 

2 

79.25674 

1.974 

0.1617 

(bandit)  *(fl6d) 

0 

2 

43.4784 

1.005 

0.3815 

X 

(sp)*(fl6c) 

-  0.202 

1 

2829.476 

130.767 

0 

X 

(sp)*(fl6d) 

-0.119 

1 

389.9447 

18.022 

0.0003 

(f!6c)*(f!6d) 

0 

1 

7.710664 

0.347 

0.5613 

After  variable  selection,  Multiple  Linear  Regression  Model  (MLRM)  technique 
and  Analysis  of  Variance  (ANOVA)  are  implemented  via  the  Minitab  statistical  package. 
The  results  and  graphics  are  discussed  next.  The  normality  assumption  is  checked  first 
because  further  analysis  is  dependent  on  this  assumption.  A  plot  of  model  residuals 
(Figure  21)  does  not  show  abnormal  departures  from  normality,  indicating  that  residuals 
meet  the  normality  assumption. 
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Normal  Probability  Plot 


Figure  21.  Normal  Probability  Plot 

Another  assumption  is  constant  variance.  This  assumption  can  be  checked  by 
plotting  residuals  versus  fitted  values  of  the  response.  If  the  plot  does  not  show  any 
pattern  like  polynomial,  outward  tunnel  or  inward  tunnel,  then  it  can  be  inferred  that 
constant  variance  assumption  is  met.  Several  residual  plots  were  examined  at  the 
stepwise  steps.  For  instance,  the  residual  plot  without  any  interaction  terms  showed  a 
polynomial  shape  which  requires  transformation  to  a  new  variable  into  the  model 
(Montgomery,  Peck,  and  Vining,  2006). 

The  residual  plot  (Figure  22)  was  obtained  by  adding  two  interaction  terms. 
There  is  no  specific  pattern  that  indicates  non  constant  variance  in  this  plot.  However,  it 
does  not  show  ideal  constant  variance.  There  is  an  accumulation  around  “100”  for  fitted 
values.  This  accumulation  originated  by  the  simulation  model  results  which  are 
unresponsive  to  some  low  and  high  values  of  input  data,  like  Bandit  numbers  and  IP 
numbers  (especially  for  the  cases  with  low  SP  numbers). 
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Figure  22.  Plot  for  Residuals  vs  Fitted  values 


Also  there  are  two  fitted  values  which  have  higher  residuals  than  others.  These 
values  belong  to  7th  and  8th  design  points  that  have  the  same  resource  numbers  except  F- 
16  D  number.  The  7th  design  point  has  six  F-16  D  and  the  8th  design  point  has  14  F-16  D. 
The  Total  Time  values  of  7th  and  8th  design  points  are  underestimated.  Their  standardized 
residuals  are  approximately  within  reasonable  range  accepted  as  between  -3  and  3  (Perry, 
2007).  Two  points  at  the  left  upper  part  of  the  Normality  Plot  (Figure  21)  and  two  points 
at  the  upper  part  of  the  Residual  Plot  (Figure  22)  belong  to  the  same  design  points. 


Table  11.  Higher  Residuals  in  DOE  Matrix 


Row 

IP 

Bandit 

SP 

F16C 

F16D 

Fitted 

Actual 

Residual 

Std.  Residual 

7 

15 

3 

28 

26 

6 

138.337 

147.402 

9.065 

2.52 

8 

15 

3 

28 

26 

14 

121.137 

132.637 

11.5 

3.16 

ANOVA  provides  a  concise  partitioning  of  variation  from  the  data  generated  by 
the  simulation  model  explained  by  the  linear  model  or  assigned  to  error  (Montgomery, 
2004).  It  also  tests  contribution  of  entire  factors  to  the  model  and  shows  whether  the 
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linear  model  is  specified  correctly  or  not.  The  ANOVA  table  (Table  12)  shows  that  all 
factors  together  are  significant  for  the  model  with  a  very  small  p  value. 


Table  12.  Analysis  of  Variance  for  Regression  Analysis 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Prob  >  F 

Model 

7 

24016.3 

3430.9 

210.98 

<.0001 

Error 

24 

390.3 

16.3 

C.  Total 

31 

24406.5 

Finally,  the  linear  model  is  formulated  to  include  the  factors  and  some  of  their 
interactions  which  are  determined  during  the  stepwise  regression  analysis.  Model 
parameter  coefficients  are  listed  at  Table  13. 


Total  Time  = 

(30  +(3 1  (IP)  +(32  (Bandit)  +(33  (SP)  +(34  (F- 1 6C)  +(35  (F- 1 6D)  +(36  (SP*F- 1 6C)  +  (37  (SP*F- 1 6D)  (2) 


As  the  coefficients  of  the  metamodel  are  considered,  is  higher  than  the  other  (3 
values  except  constant  coefficient  ((3o).  The  SP  factor  previously  showed  up  as  a 
significant  factor  for  the  simulation  model  after  analyzing  the  output  data  via  DOE. 
Additionally  P4  is  the  next  higher  coefficient  corresponding  to  the  F-16C  factor.  This  also 
shows  a  parallelism  with  the  analysis  of  main  effects  and  stepwise  selection.  The  F-16D 
factor,  which  is  less  significant  since  the  SPs  are  scheduled  to  fly  more  solo  missions 
during  the  flight  training  term,  follows  the  F-16C  factor.  The  last  two  main  factors,  IP 
and  Bandit,  have  minimum  coefficients  which  coincide  with  the  comments  previously 
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made  related  to  the  utilization  perspective.  Also  correlation  between  every  term, 
including  Total  Time,  was  analyzed.  The  correlation  between  SP  and  Total  Time  is  0.81 
and  is  the  highest  positive  correlation.  The  correlation  table  is  placed  in  Appendix  F. 


Table  13.  Parameter  coefficients 


Coefficient 

Po 

19.3 

Pi 

-0.244 

p2 

-0.251 

p3 

8.01 

P4 

2.24 

P5 

1.2 

P6 

-0.2 

P7 

-0.119 

To  check  the  accuracy  of  our  metamodel,  ten  random  resource  levels  are  selected 
and  both  simulation  model  and  linear  model  are  run  for  these  ten  samples.  For  the 
simulation  model  results,  95  %  confidence  intervals  are  calculated  to  see  whether  our 
metamodel’s  results  are  covered  by  these  intervals.  Results  are  shown  at  Table  14  with 
those  cases  where  the  models  are  not  significantly  different  in  bold. 
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Table  14.  Results  from  the  simulation  model  and  the  metamodel 


IP 

Bandit 

SP 

F-16C 

F-16D 

Simulation 

95  %  Confidence 
Interval 

Model 

1 

15 

6 

13 

24 

6 

109.17 

106.886 

111.454 

107.542 

2 

15 

9 

15 

23 

10 

108.06 

105.75 

110.37 

110.201 

3 

23 

7 

21 

21 

12 

115.06 

113.034 

117.086 

123.393 

4 

24 

8 

20 

24 

9 

113.89 

112.1794 

115.6006 

118.776 

5 

28 

8 

15 

23 

11 

106.93 

105.05 

108.81 

106.695 

6 

28 

8 

23 

25 

14 

113.1 

111.221 

114.979 

114.172 

7 

21 

6 

19 

20 

10 

122.67 

120.21 

125.13 

123.05 

8 

21 

5 

21 

23 

9 

126.73 

124.22 

129.24 

124.36 

9 

18 

9 

13 

20 

8 

105.27 

103.48 

107.05 

106.803 

10 

19 

8 

25 

26 

9 

122.9 

121.01 

124.79 

125.171 

Seventy  percent  of  our  metamodel  results  are  covered  by  the  confidence  intervals 
from  the  simulation.  Figure  23  displays  both  results  together.  The  three  values  not 
captured  by  the  simulation  confidence  intervals  are  higher  than  the  simulation  results. 
The  metamodel  overestimated  those  training  periods  with  the  given  resource  and  SP 
levels.  The  values  remain  approximately  three  standard  deviations  away  from  the 
simulation  results. 


Total  Time  (Simulation  &  Metamodel) 


□  Simulation 
■  Metamodel 


123456789  10 

Sample  Number 


Figure  23.  Total  Times  of  Training  for  Two  models 
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4.3.  Sensitivity  Analysis  for  Recent  Resource  Numbers 

One  of  the  main  focuses  of  this  study  was  to  have  an  idea  about  acceptable  limits 
of  the  system  for  processing  increasing  number  of  SPs  with  various  resource  levels.  This 
type  of  tool  provides  the  decision  maker  with  an  insight  about  system  reaction  when  each 
resource  level  is  separately  increased  or  decreased.  Specifically  it  may  also  provide  an 
answer  to  the  question  whether  the  squadron  can  provide  training  to  an  expected  number 
of  SPs  within  the  acceptable  Total  Time.  The  recent  resource  numbers  from  the  143rd  F- 
16  Pilot  Training  Squadron  are  given  at  Table  15. 

Table  15.  Numbers  of  Resources 


Resources 

Number 

IP 

18 

Bandit 

10 

F-16C 

20 

F-16D 

8 

143rd  Oncel  squadron  completes  the  training  in  approximately  103  weekdays  via 
the  metamodel  and  102  weekdays  via  simulation  model  with  the  given  resource  levels 
and  12  SPs.  With  these  resource  and  SP  numbers,  the  actual  training  lasted  99  weekdays. 
The  analysis  was  conducted  both  by  using  the  metamodel  and  the  simulation  model  in 
Process  Analyzer.  Process  Analyzer  results  for  the  simulation  model  are  shown  in 
Appendix  G. 
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Since  IPs  and  Bandits  can  fly  twice  in  a  day,  these  factors  do  not  significantly 
contribute  to  Total  Time  as  much  as  the  other  factors.  As  stated  early  their  utilization 
values  (working  hours),  given  at  Table  16,  are  increased.  For  this  reason  the  factors 
including  F-16  C,  F-16  D  and  SP  number  will  be  examined  in  this  analysis. 


Table  16.  Utilizations  depending  on  resource  levels 


IP 

Bandit 

SP 

F-16C 

F-16D 

TIME 

IP  util. 

Bandit  util. 

15 

3 

12 

14 

6 

103.37 

0.724 

0.898 

15 

3 

12 

14 

14 

104.104 

0.731 

0.874 

15 

3 

12 

26 

6 

103.768 

0.724 

0.878 

15 

3 

12 

26 

14 

100.903 

0.738 

0.861 

15 

3 

28 

14 

6 

175.868 

0.712 

0.831 

15 

3 

28 

14 

14 

161.968 

0.788 

0.869 

15 

3 

28 

26 

6 

147.402 

0.829 

0.923 

15 

3 

28 

26 

14 

132.637 

0.935 

0.97 

15 

11 

12 

14 

6 

105.769 

0.649 

0.381 

15 

11 

12 

14 

14 

102.346 

0.661 

0.378 

15 

11 

12 

26 

6 

101.97 

0.655 

0.384 

15 

11 

12 

26 

14 

99.004 

0.662 

0.374 

15 

11 

28 

14 

6 

175.235 

0.619 

0.378 

15 

11 

28 

14 

14 

160.569 

0.698 

0.407 

15 

11 

28 

26 

6 

135.135 

0.754 

0.471 

15 

11 

28 

26 

14 

118.502 

0.895 

0.51 

31 

3 

12 

14 

6 

103.335 

0.278 

0.86 

31 

3 

12 

14 

14 

103.142 

0.277 

0.82 

31 

3 

12 

26 

6 

102.103 

0.281 

0.824 

31 

3 

12 

26 

14 

100.336 

0.297 

0.772 

31 

3 

28 

14 

6 

174.77 

0.315 

0.781 

31 

3 

28 

14 

14 

157.969 

0.354 

0.822 

31 

3 

28 

26 

6 

131.437 

0.401 

0.891 

31 

3 

28 

26 

14 

110.302 

0.443 

0.948 

31 

11 

12 

14 

6 

103.969 

0.245 

0.367 

31 

11 

12 

14 

14 

101.001 

0.249 

0.371 

31 

11 

12 

26 

6 

102.209 

0.253 

0.357 

31 

11 

12 

26 

14 

100.835 

0.26 

0.352 

31 

11 

28 

14 

6 

175.101 

0.279 

0.36 

31 

11 

28 

14 

14 

158.568 

0.309 

0.385 

31 

11 

28 

26 

6 

131.201 

0.349 

0.446 

31 

11 

28 

26 

14 

109.435 

0.376 

0.476 

67 


First  of  all,  the  question  of  how  many  additional  SPs  the  143rd  Oncel  squadron 
can  handle  by  using  the  same  resource  levels  within  the  acceptable  Total  Time  will  be 


examined.  As  mentioned  in  previous  chapters,  there  are  two  consecutive  training  periods 
which  may  affect  each  other  if  any  extension  occurs  during  the  training  Total  Time.  The 
acceptable  delay  criteria  is  defined  to  see  how  many  additional  SPs  can  be  trained  within 
the  acceptable  time  limits  while  avoiding  a  major  delay  in  the  training  period.  The 
acceptable  delay  is  determined  as  ten  weekdays  for  the  number  of  resources  given  in 
Table  15.  It  is  expected  that  each  additional  SP  increases  the  Total  Time.  The  same 
calculation  is  done  by  using  the  simulation  model  and  metamodel. 

After  adding  one  more  SP  to  the  model,  the  new  Total  Time  values  are  recorded 
and  shown  at  Table  17.  As  a  result,  each  additional  SP  increases  the  Total  Time  by 
approximately  three  weekdays  for  the  metamodel  and  two  weekdays  for  the  simulation 
model.  In  this  case,  an  additional  three  SPs  can  be  added  to  the  training  program  and  the 
Total  Time  will  extend  approximately  nine  weekdays.  Even  when  the  SP  number  is 
doubled,  the  Total  Time  increased  with  the  same  proportion.  The  system  could  handle 
double  the  number  of  SPs  within  six  months  due  to  the  current  low  number  of  SP  and 
sufficient  number  of  other  resources. 


Table  17.  Effects  of  Each  Additional  SP 


Additional  SP 

Simulation 

Model 

0 

101.73 

102.86 

1 

103.967 

106.82 

2 

106.436 

109.82 

3 

108.875 

112.82 
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Secondly  the  F-16  C  effect  to  the  training  period  will  be  considered.  Each  F-16  C 


reduction  expectedly  increases  the  Total  Time.  The  new  Total  Time  values,  as 
determined  by  simulation  and  metamodel,  are  shown  in  Table  18  corresponding  to  each 
reduction  in  F-16  C  number.  Each  F-16  C  model  reduction  increases  the  Total  Time  by 
approximately  one  weekday  for  the  metamodel  and  approximately  two  weekdays  (on 
average)  for  the  simulation  model.  Total  Time  does  not  extend  past  ten  weekdays  even  if 
the  squadron  faces  5-unit  F-16  C  reductions  for  the  simulation  model.  These  results  show 
that  current  number  of  F-16  C  is  higher  than  needed.  Since  SPs  are  not  allowed  to  fly 
more  than  one  sortie  and  there  are  less  numbers  of  SPs,  each  reduction  in  F-16  Cs  did  not 
increase  the  Total  Time  significantly. 


Table  18.  Effect  of  Each  Reduction  in  F-16  C  on  Total  Time 


F-16  C  Reduction 

Simulation 

Model 

0 

101.73 

102.86 

1 

102.703 

104.008 

2 

110.067 

104.196 

3 

109.402 

104.384 

4 

109.736 

104.572 

The  same  steps  are  implemented  for  the  F-16  D  factor.  The  metamodel  does  not 
care  whether  one  of  the  resources  is  zero  or  not  since  it  is  a  linear  formulation.  The 
simulation  model  normally  does  not  run  under  these  conditions.  If  F-16  D  number  was 
inserted  unrealistically  as  “0”,  the  simulation  model  would  give  an  error.  Each  F-16  D 
reduction  in  the  simulation  model,  especially  in  low  F-16  D  numbers,  did  not  increase  the 
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Total  Time  linearly.  Therefore,  our  analysis  only  considers  the  simulation  results  under 
these  circumstances.  The  simulation  model  results  are  shown  at  Table  19. 


Table  19.  Effect  of  Each  Reduction  in  F-16  D  on  Total  Time 


F-16  D  Reduction 

Simulation 

0 

101.73 

1 

109.671 

2 

109.269 

3 

112.635 

4 

121.509 

As  a  summary,  this  analysis  showed  that  the  squadron  can  handle  more  SPs  than 
the  current  number  within  acceptable  extension  in  the  Total  Time  with  the  given 
resources.  Even  when  the  SP  number  was  doubled,  the  Total  Training  period  lasted 
approximately  6  months.  However,  utilization  values  were  increased  accordingly.  The 
analysis  results  obviously  depend  on  specified  factor  numbers.  Therefore  this  case  study 
reflects  the  results  based  on  previously  specified  resource  levels  (Table  15).  We  also 
examined  different  scenarios  with  other  resource  levels  that  showed  similar  results. 

4.4.  Conclusion 

In  this  chapter  the  data  produced  by  the  simulation  model  runs  are  analyzed  to 
have  an  idea  how  the  system  reacts  with  different  levels  of  resources  and  entities.  As  a 
summary,  regression  analysis  techniques  and  metamodeling  are  implemented  through  this 
chapter.  Additionally  as  visual  presentation,  main  effects,  interactions  between  factors 
and  cube  plots  were  used. 
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As  stated  in  Chapter  3,  there  are  two  flight  training  terms  in  a  year.  The  results 
and  following  analysis  focus  only  on  the  first  term  which  is  held  during  the  first  6  months 
of  the  year.  The  second  term  analysis  is  redundant  of  the  first  and  therefore  not  discussed. 


71 


V.  Conclusions  and  Recommendations 


General 

This  chapter  reviews  the  key  points  of  this  study.  The  previous  four  chapters 
formed  the  foundation  of  this  simulation  study.  First  a  short  summary  of  these  chapters 
will  be  mentioned.  Following  the  summary,  recommendations  for  further  research  topics 
will  be  discussed. 

5.1.  Research  and  Conclusions 

The  simulation  study  of  the  F- 16  Pilot  Training  Squadron  was  initiated  first  by 
defining  the  main  interest  and  describing  the  organization  and  procedures  of  the  flight 
training.  The  additional  ideas,  extending  the  coverage  of  this  study,  emerged  during  the 
construction  phase  of  the  simulation  model.  Subsequently  design  of  experiment  was 
implemented  to  collect  necessary  output  data  from  our  simulation  model  for  following 
analysis.  The  results  were  presented  in  Chapter  4  using  statistical  analysis  techniques. 

The  flight  phase  which  is  the  most  important  phase  of  the  entire  training  period 
was  captured  within  the  simulation  model.  The  final  simulation  model  does  not  reflect  the 
real  world  exactly  due  to  the  following  reasons.  First  of  all  the  maintenance  factors  were 
not  embedded  in  the  Arena  model.  Only  spare  aircraft  policy  was  generally  covered.  Like 
the  maintenance  factor,  the  personnel  medical  status  (i.e.,  sickness)  was  not  included  due 
to  lack  of  sufficient  data.  Lastly,  the  weather  factor  was  embedded  into  the  simulation 
model  by  only  considering  daily  forecast.  It  is  obvious  that  the  weather  status  may 
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change  temporarily,  for  instance  the  first  block  of  flights  may  be  cancelled  due  to  weather 
conditions,  but  the  second  block  of  flights  may  continue.  The  aim  was  to  come  up  with  a 
model  which  is  considered  close  enough  to  the  real  system. 

In  the  analysis  phase,  system  reaction  on  under  certain  conditions  such  as  SP 
number  increase,  F-16  C  and  F-16  D  decrease  was  investigated.  Sensitivity  analysis  was 
also  conducted.  It  was  realized  that  SPs  are  the  most  significant  factor  in  the  system. 
Although  IPs  and  Bandits  are  main  players  of  the  flight  training,  they  unexpectedly  did 
not  contribute  to  the  model  as  much  as  the  other  three  factors  due  to  their  (IPs-Bandits) 
utilization  factor.  Unlike  SPs,  IPs  and  Bandits  may  be  required  to  fly  more  than  one  sortie 
in  a  day  resulting  in  an  increase  in  utilization  values  (work  hours).  Also  it  was  observed 
that  the  F-16  C  is  more  significant  than  the  F-16  D.  As  stated  in  the  training  syllabus,  the 
solo  sorties  requiring  F-16  C  allocation  are  more  than  the  dual  sorties  requiring  F-16  D 
allocation.  It  is  considered  that  this  allocation  makes  the  F-16  C  more  significant  in  the 
simulation  model.  The  same  fact  was  found  via  statistical  analysis  reflected  as  a  higher 
coefficient  in  the  metamodel  and  steeper  slope  in  the  main  effect  plots  versus  F-16  D. 

By  using  the  simulation  model,  a  metamodel  was  constructed  via  multiple  linear 
regression  analysis.  The  interactions  between  factors  were  tested.  It  was  observed  that  the 
interaction  terms  between  SP  and  each  F-16  type  significantly  contributed  to  the 
metamodel.  Subsequently  the  sensitivity  analysis  was  implemented  for  the  given  sample 
resource  levels. 

Each  additional  SP  increased  the  Total  Time  more  than  the  other  factors.  Each 
additional  SP  resulted  in  approximately  two-weekday  additional  time  to  the  Total  Time 
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based  on  the  recent  resource  levels.  Also  each  additional  decrease  in  F-16  C  number 
resulted  in  approximately  one- weekday  extension.  The  F-16  D  reduction  affected  the 
system  in  a  non-linear  fashion.  These  results  vary  depending  on  the  given  SP  and 
resource  levels. 

5.2.  Recommendations 

The  simulation  model  can  be  improved  by  adding  logic  which  more  accurately 
considers  the  utilization  of  IPs  and  Bandits.  In  the  current  model,  utilization  values  are 
collected  as  a  subsidiary  performance  measures  to  observe  how  they  tend  to  vary.  The 
utilization  values  can  be  embedded  into  the  simulation  model  to  limit  the  daily  or  weekly 
work  hours  for  IPs  and  Bandits. 

If  the  related  statistical  information  is  obtained,  the  daily  medical  status  can  be 
reflected  in  the  simulation  model.  A  medical  effect  to  the  training  periods  can  then  be 
analyzed  to  see  how  significant  it  is  in  the  model.  Also  the  weather  factor  per  each  block 
can  be  embedded  into  the  model.  This  extension  makes  the  simulation  more  realistic  than 
the  current  one.  But  this  addition  requires  detailed  information  corresponding  to  each 
block. 

Another  contribution  to  the  simulation  model  can  be  made  in  the  maintenance 
perspective.  Maintenance  effect  on  the  current  operative  F-16  number  may  vary 
depending  on  many  factors  like  weather,  periodic  maintenance,  number  of  the 
maintenance  staff,  spare  aircraft,  and  so  forth.  First  only  the  flight  training  period  was  the 
point  of  interest  of  this  study  according  to  procedures  stated  in  the  syllabus.  The 
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maintenance  can  be  modeled  as  a  separate  part  and  embedded  to  the  flight  training 
simulation  model. 


75 


Appendix  A.  List  of  Abbreviations 


AA 

Air  to  Air 

AG 

Air  to  Ground 

ACM 

Air  Combat  Maneuvering 

AREC 

Armed  Reconnaissance 

BFM 

Basic  Fighter  Maneuvering 

BTR 

Basic  Training 

DOE 

Design  of  Experiment 

INT 

Intercept 

IP 

Instructor  Pilot 

IQ 

Instrument  Qualification 

MLRM 

Multiple  Linear  Regression  Model 

NI 

Night  Intercept 

NTR 

Night  Training 

RSU 

Runway  Supervisor  Unit 

SA 

Surface  Attack 

SAT 

Tactical  Surface  Attack 

SP 

Student  Pilot 

STA 

Surface  Tactical  Attack 

TQ 

Tactical  Qualification 
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Appendix  C.  Factors  and  Responses  from  DOE 


IP 

Bandit 

SP 

F-16C 

F-16D 

TIME 

IP  util. 

Bandit  util 

15 

3 

12 

14 

6 

103.37 

0.724 

0.898 

15 

3 

12 

14 

14 

104.104 

0.731 

0.874 

15 

3 

12 

26 

6 

103.768 

0.724 

0.878 

15 

3 

12 

26 

14 

100.903 

0.738 

0.861 

15 

3 

28 

14 

6 

175.868 

0.712 

0.831 

15 

3 

28 

14 

14 

161.968 

0.788 

0.869 

15 

3 

28 

26 

6 

147.402 

0.829 

0.923 

15 

3 

28 

26 

14 

132.637 

0.935 

0.97 

15 

11 

12 

14 

6 

105.769 

0.649 

0.381 

15 

11 

12 

14 

14 

102.346 

0.661 

0.378 

15 

11 

12 

26 

6 

101.97 

0.655 

0.384 

15 

11 

12 

26 

14 

99.004 

0.662 

0.374 

15 

11 

28 

14 

6 

175.235 

0.619 

0.378 

15 

11 

28 

14 

14 

160.569 

0.698 

0.407 

15 

11 

28 

26 

6 

135.135 

0.754 

0.471 

15 

11 

28 

26 

14 

118.502 

0.895 

0.51 

31 

3 

12 

14 

6 

103.335 

0.278 

0.86 

31 

3 

12 

14 

14 

103.142 

0.277 

0.82 

31 

3 

12 

26 

6 

102.103 

0.281 

0.824 

31 

3 

12 

26 

14 

100.336 

0.297 

0.772 

31 

3 

28 

14 

6 

174.77 

0.315 

0.781 

31 

3 

28 

14 

14 

157.969 

0.354 

0.822 

31 

3 

28 

26 

6 

131.437 

0.401 

0.891 

31 

3 

28 

26 

14 

110.302 

0.443 

0.948 

31 

11 

12 

14 

6 

103.969 

0.245 

0.367 

31 

11 

12 

14 

14 

101.001 

0.249 

0.371 

31 

11 

12 

26 

6 

102.209 

0.253 

0.357 

31 

11 

12 

26 

14 

100.835 

0.26 

0.352 

31 

11 

28 

14 

6 

175.101 

0.279 

0.36 

31 

11 

28 

14 

14 

158.568 

0.309 

0.385 

31 

11 

28 

26 

6 

131.201 

0.349 

0.446 

31 

11 

28 

26 

14 

109.435 

0.376 

0.476 

Appendix  D.  Main  Effect  Plots  for  IP  and  Bandit  Utilizations 


Main  Effects  Plot  for  IP  utilization 


Main  Effects  Plot  for  Bandit  utilization 
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Appendix  E.  Step  History 


Step 

Parameter 

Action 

"Sig 

Seq  SS 

RSquare 

Cp 

1 

ip 

Entered 

0.6563 

156.1851 

0.0067 

1430.4 

2 

bandit 

Entered 

0.8297 

37.6148 

0.0083 

1430 

3 

sp 

Entered 

0 

15369.69 

0.6666 

465.53 

4 

fl6c 

Entered 

0.0002 

3206.003 

0.8039 

265.92 

5 

fl6d 

Entered 

0.0216 

855.8419 

0.8406 

214.1 

6 

(ip)*(bandit) 

Entered 

0.676 

26.44099 

0.8417 

214.44 

7 

(ip)*(bandit) 

Removed 

0.676 

26.44099 

0.8406 

214.1 

8 

(ip)*(sp) 

Entered 

0.4281 

94.17409 

0.8446 

210.18 

9 

(ip)*(sp) 

Removed 

0.4281 

94.17409 

0.8406 

214.1 

10 

(ip)*(fl6c) 

Entered 

0.6037 

40.71178 

0.8423 

213.54 

11 

(ip)*(fl6c) 

Removed 

0.6037 

40.71178 

0.8406 

214.1 

12 

(ip)*(fl6d) 

Entered 

0.7292 

18.16236 

0.8413 

214.96 

13 

(ip)*(fl6d) 

Removed 

0.7292 

18.16236 

0.8406 

214.1 

14 

(bandit)*(sp) 

Entered 

0.7597 

14.18048 

0.8412 

215.21 

15 

(bandit)  *(sp) 

Removed 

0.7597 

14.18048 

0.8406 

214.1 

16 

(bandit)*(fl6c) 

Entered 

0.5996 

41.64194 

0.8423 

213.48 

17 

(bandit)*(fl6c) 

Removed 

0.5996 

41.64194 

0.8406 

214.1 

18 

(bandit)  *(fl6d) 

Entered 

0.8442 

5.8636 

0.8408 

215.73 

19 

(bandit)  *(fl6d) 

Removed 

0.8442 

5.8636 

0.8406 

214.1 

20 

(sp)*(fl6c) 

Entered 

0 

2829.476 

0.9617 

38.172 

21 

(sp)*(fl6d) 

Entered 

0.0002 

389.9447 

0.9784 

15.651 

22 

(fl6c)*(fl6d) 

Entered 

0.5556 

7.710664 

0.9788 

17.166 

23 

(fl6c)*(fl6d) 

Removed 

0.5556 

7.710664 

0.9784 

15.651 

24 

bandit 

Removed 

0.193 

37.6148 

0.9768 

16.016 

25 

ip 

Removed 

0.0126 

156.1851 

0.9701 

23.838 

26 

ip 

Entered 

0.0126 

156.1851 

0.9768 

16.016 

80 


Appendix  F.  Correlations  between  Metamodel  Factors 


IP 

Bandit 

SP 

F-16C 

F-16D 

SPT-16C 

SPT-16D 

SP*IP 

TIME 

IP 

1 

Bandit 

0 

1 

SP 

0 

0 

1 

F-16C 

0 

0 

0 

1 

F-16D 

0 

0 

0 

0 

1 

SP*F-16C 

0 

0 

0.777 

0.583 

0 

1 

SPT-16D 

0 

0 

0.680 

0 

0.680 

0.529 

1 

SP*IP 

0.634 

0 

0.729 

0 

0 

0.567 

0.496 

1 

TIME 

-0.071 

-0.036 

0.812 

-0.384 

-0.170 

0.326 

0.399 

0.532 

1 
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Appendix  G.  Process  Analyzer  Results  for  Sensitivity  Analysis 
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